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FOREWORD 



America’s workers are facing their biggest challenge in a generation. They must have the skills 
and knowledge, and the flexibility to use them, to help American firms create goods and services 
that can compete with those from other nations. In response to this need, the nation’s governors 
in 1989, and later the U.S. Congress though the Goals 2000 legislation, outlined a set of Na- 
tional Education Goals. The ultimate objective of the National Education Goals was enhance- 
ment of learning in this nation. Goal 6, Objective 5, focused upon the higher order thinking and 
communication skills needed by college graduates, in the workplace and for the practice of citi- 
zenship. 

The National Center for Education Statistics (NCES) has been concerned with the assessment of 
student learning in the nation’s schools and colleges. Over the past 30 years, it has assessed stu- 
dent learning at the K-12 level. In preparation for the assessment of student learning at the col- 
lege level, NCES has conducted three workshops over the past four years. The first identified 
issues and concerns related to assessment of college student learning, the second was intended to 
identify more clearly the skills and sub-skills needed, while the third was focused upon state as- 
sessment activities in 1996 with special emphasis on the skills cited in the National Education 
Goals. This publication reports on the results of the third workshop. 

The workshop was held in Arlington, Virginia, on December 7-8, 1995. It was conducted in co- 
operation with the Education Commission of the States, the State Higher Education Executive 
Officers Organization, and the National Center for Postsecondary Teaching, Learning, and As- 
sessment. Workshop proceedings were compiled by Peter Ewell of the National Center for 
Higher Education Management Systems. The workshop conference contractor was Professional 
and Scientific Associates of McLean, Virginia, ably represented by Regina Guyther. Sal Cor- 
rallo, Project Director for the National Assessment of College Student Learning, was conference 
coordinator. Our thanks go to all who attended and contributed, especially the state participants 
who prepared the background papers and other information. 

Pascal D. Forgione, Jr. 

Commissioner, NCES 
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THE NATIONAL ASSESSMENT OF COLLEGE STUDENT LEARNING: 
AN INVENTORY OF STATE-LEVEL ASSESSMENT ACTIVITIES 



A REPORT OF THE PROCEEDINGS OF THE 
THIRD STUDY DESIGN WORKSHOP 

I. Background and Overview 

This document summarizes proceedings and conclusions of a two-day national planning work- 
shop on the topic of furthering the assessment of national postsecondary outcomes, sponsored by 
the National Center for Education Statistics (NCES) and held in Washington, D.C. on December 
7-8, 1995. Activities at the workshop were designed specifically to address the degree to which 
state-level assessment initiatives in higher education might aid in the construction of a national in- 
dicator of postsecondary attainment consistent with Goal 6.5 of the National Education Goals, 
and to determine ways in which NCES and the states might work more effectively to develop mu- 
tually-supporting activities and policies in the realm of postsecondary assessment. As a conse- 
quence, invited participants consisted of representatives from each state, selected assessment and 
national policy experts, and NCES staff. Conclusions of the workshop indicate that there is no im- 
mediate possibility of aggregating existing state-level data on postsecondary outcomes to create a 
usable national indicator. But they do suggest a number of other ways in which the states and the 
federal government might work together to improve the quality of data available on this topic. 

A. Background 

The topic of assessing the outcomes of postsecondary education emerged as an area of con- 
siderable policy concern in the early 1980’s. Like its counterpart in K-12 education, the 
topic was fueled by concerns about performance, a growing need for accountability related 
to increasing levels of public investment and above all, an interest in stimulating higher lev- 
els of quality. A powerful potential impetus for both conversations was the National Educa- 
tion Goals agreed upon by former President Bush and the nation’s governors in 1989. 

Though chiefly concerned with the improvement of elementary and secondary education, 
outcomes of postsecondary education were explicitly addressed by the Goals. In particular, 
Goal 6, Objective 5 of both the original document and its current embodiment in President 
Clinton’s "Goals 2000" requires that "the proportion of college graduates who demonstrate 
an advanced ability to think critically, communicate effectively, and solve problems will in- 
crease substantially." Goal 3, Objective 2 also contains references to communications and 
thinking skills, along with several other subjects, at both the K-12 and postsecondary levels. 

In the four years subsequent to the establishment of the National Education Goals, NCES 
conducted two planning workshops and funded a number of studies intended to develop a 
means to assess postsecondary learning consistent with Goal 6.5 (see Appendix A-3). The 
first study design workshop, held in 1991, was designed to identify the specific issues and 
concerns that might surround such an assessment, and involved a wide cross-section of do- 
main and assessment expertise. A second, held in 1992, began the process of specifying the 
required assessment domain by identifying specific skills, knowledge, and abilities associ- 
ated with the attainment of the abilities noted under Goal 6.5. Results of the workshops led 



to the development of a formal Request for Proposal (RFP) that focused on further skills- 
specification suitable in detail and form to guide the development of a national survey of 
collegiate learning. That RFP, however, was later withdrawn due to lack of funds. 

This action, however, has not diminished the demand for information from policymakers, 
employers, and the general public about how well college graduates are prepared to exercise 
workplace skills and to fulfill the obligations of citizenship. To help meet this demand, 
NCES attempted to identify alternative approaches to gathering information on national col- 
legiate attainment. To that end, a number of more limited studies were sponsored. These in- 
cluded a Delphi study by the National Center for Postsecondary Teaching Learning and 
Assessment (NCTLA) at Penn State designed to provide an initial listing of the domain, a 
National Job Analysis study intended to define a generic set of job skills in high-perform- 
ance workplace settings, and completion of the National Adult Literacy Study (NALS) in 
1994 which contained a restricted non-systematic sample of college graduates of all ages. 

Given the limits of these activities in actually generating data on collegiate attainment, 
NCES decided to turn to the states for assistance. By 1989, it was known that some two- 
thirds of the states had developed assessment policies governing their public higher educa- 
tion systems (Ewell, Finney and Lenth 1990). No systematic analysis of state-level 
assessment activities in postsecondary assessment had been conducted since 1989, however. 
NCES was therefore interested in determining both the current extent and character of such 
activities and, more particularly, the degree to which their results might be able to provide 
an insight into the progress being made nationally toward the achievement of Goal 6.5. Be- 
cause it was expected that states would differ in both capacity and approach, the posed ques- 
tion was not whether raw state results could be summarized into a single indicator. Rather it 
was the degree to which these many state efforts might help to paint a broader collective pic- 
ture of collegiate attainment in relation to broadly-identified workplace and societal skills. 

B. Organization of the Workshop 

The workshop brought together representatives responsible for postsecondary assessment 
activities in the fifty states and selected territories, with assessment experts and selected 
NCES staff (see the participants list included as Appendix A-2). Its design included both a 
set of pre-workshop inventorying activities and a working agenda intended to address four 
questions: 

• what specific areas of knowledge, skills, and attributes are being assessed at the state 
level, and what is their commonality both across states and with the skills identified in 
Goal 6.5? 

• how are these outcomes being assessed, with what frequency are they being reported, and 
how are the results being used? 

• what might be done to help states broaden and enhance their assessment efforts in ways 
that would benefit both the states and meet the need for national reporting? 

• what is the appropriate role of NCES in assisting states and institutions to gather and re- 
port better information about postsecondary results? 




2 



8 



Pre-workshop activities were of several kinds. First, in order to provide an updated sum- 
mary of current state postsecondary assessment initiatives, states were requested to prepare 
background papers describing their current activities, and to bring these to the workshop 
(see Section II below). Second, a number of additional background papers were commis- 
sioned or assembled for prior distribution to participants and/or for delivery at the work- 
shop. These included a report by ACT on state and institutional assessment needs, brief case 
studies of assessment efforts and related activities in several states, and summaries of assess- 
ment instruments and approaches by researchers and assessment organizations. Selected pa- 
pers in this series are provided as Appendix C of these Proceedings. 

The workshop agenda itself consisted of a mix of plenary and working sessions (see Appen- 
dix A-l). Plenary sessions were organized around three broad topics — current state assess- 
ment activities and approaches, the political context for assessment, and methodological 
considerations involved in postsecondary assessment. Working sessions on Thursday eve- 
ning and Friday morning were intended to explore respectively the current potential of state 
data-collection efforts to inform national discussions of collegiate attainment, and the kinds 
of actions that might be taken at both the state and federal levels to further postsecondary as- 
sessment efforts. The workshop ended with a final plenary designed to advance and refine 
some tentative conclusions. 

C. Organization of the Proceedings 

The text of these Proceedings follows the logic of the workshop itself, with supporting docu- 
ments and background papers provided as Appendices. Section II presents results of the in- 
ventory of state activities conducted as part of the workshop, and is intended to update 
collective understanding of current state-level postsecondary assessment activities. As such, 
its publication and dissemination constitutes one important intended outcome of the meet- 
ing. Section III describes the major themes that emerged from workshop discussions — both 
in the plenary and provided by the various working groups. These are discussed under three 
main subheadings — the current political context for postsecondary assessment, the corre- 
sponding methodological context, and specific actions that might be taken. Section IV pro- 
vides a summary of possible next steps about which consensus emerged for both the states 
and NCES. Appendices to the document include background materials for the workshop 
(Appendix A), the full texts of all state background papers prepared as part of the workshop 
(Appendix B), and the texts of selected additional background papers prepared by partici- 
pants (Appendix C). 

II. Inventory of State Assessment Activities 

An important prerequisite of the workshop was to determine the extent and character of current 
state-level activities in assessing postsecondary student outcomes. Accordingly, each state was re- 
quested to inventory such activities in the form of a background paper to be shared at the work- 
shop. More particularly, the papers were intended to determine the degree to which the results of 
such activities are sufficiently consistent with one another and with the domains addressed by 
Goal 6.5 to provide an initial basis for constructing a national indicator of collegiate achieve- 
ment. In this respect, the information provided by the state background papers indicated that cur- 
rent state programs could not provide such a basis. States vary considerably in their approaches 
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and policies, and state assessment programs as a body do not provide a foundation for meaning- 
ful national reporting. But the papers also revealed many themes and problems in common across 
states that were useful to participants in thinking about collective needs and about how federal 
and state actors in the realm of assessment might work together more effectively. 

The resulting summary of state assessment activities built upon two previous national inventories 
of this kind. The first such fifty-state inventory was compiled by the Education Commission of 
the States (Boyer et. al. 1987), and took place when states were just beginning to implement for- 
mal policies on assessing collegiate achievement. At that point, some fifteen states had estab- 
lished visible programs and more than half were exploring the issue — though few contemplated 
deploying the kinds of large-scale assessments using standardized instruments that were then 
common in K-12 education. The second such inventory was compiled three years later by the 
Education Commission of the States and the State Higher Education Executive Officers (Ewell, 
Finney and Lenth 1990). By this point, more than two-thirds of the states had adopted or were de- 
veloping formal assessment policies, and only nine indicated that they had no plans of this kind. 
Again, however, very few standardized assessment methods were mandated and most such poli- 
cies required institutions to develop their own assessment approaches consistent with local mis- 
sions and student clienteles. 

The 1 995 inventory conducted through the workshop was intended partly to update the informa- 
tion obtained through these prior efforts. More specifically, it was aimed at determining areas of 
potential common interest and the alignment of current state activities with the outcomes do- 
mains associated with Goal 6.5 — the abilities to "think critically, communicate effectively, and 
solve problems." To that end, the state background papers consisted of two components. First, 
states were requested to describe the origins and development of their approach and, more specifi- 
cally, to indicate the kinds of instruments employed and their potential for contributing informa- 
tion useful to a national indicator of collegiate achievement. The full texts of all state 
submissions under this heading is included as Appendix B. Secondly, states were asked to iden- 
tify important obstacles to furthering their assessment initiatives, important methodological prob- 
lems involved in developing assessment approaches, and what they saw as the most important 
needs to move a national assessment agenda forward. 

In all, thirty-four states and Puerto Rico prepared background papers as requested. While the for- 
mal response was less than that achieved by the prior two inventories, a combination of follow- 
up efforts and a review of available published documents allowed an accurate basic inventory of 
activity to be constructed for all fifty states and Puerto Rico. Papers were prepared largely by 
SHEEO’s or their designees, and the majority attached additional documentation describing their 
policies, approaches, and results obtained. Contextual information was also supplied by two addi- 
tional studies conducted by other agencies. The first was a SHEEO survey of state-level manage- 
ment information system capabilities which yielded a positive picture of the evolution of 
state-level technical capacity in the area of student information (Russell and Chisholm 1995). 

The second was an interview-based study of education leaders in forty-nine state legislatures con- 
ducted by the National Education Association, which provided useful insights into the shifting 
political context of accountability policy in higher education (Ruppert 1996). 
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A. The Current Status of State Assessment Activity 



Results of the 1995 inventory suggest that little of substance has changed with respect to 
state-level assessment since 1989. More states are now involved in the activity, but the addi- 
tions are those which had previously indicated plans for doing so. Most states, moreover, are 
continuing to support an "institution-centered" approach that emphasizes local development 
and use of results and contains no common measures. But results of the 1995 inventory also 
suggest a wider political context for assessment significantly different from that apparent in 
1989. Especially salient here is a growing policy preoccupation with governance and fiscal 
matters at the state level, effectively displacing the agenda of undergraduate educational im- 
provement that originally gave rise to both assessment and the National Goals movement in 
the eighties. 

1. Policy Patterns. Chart 1 presents the current status of the states with respect to 



postsecondary assessment initiatives and describes these initiatives across a num- 
ber of policy categories. About half the states are maintaining the "institution-cen- 
tered" policy approach pioneered by such states as Virginia and Colorado about a 
decade ago. This approach emphasizes the development of institution-level assess- 
ment methods that best fit the mission and student clientele of each individual col- 
lege or university. Most are governed by state-level guidelines for the 
development of assessment measures, but require no commonality across institu- 
tions. Most also require periodic public reporting of results by institutions either 
annually or biennially. About two-thirds of such initiatives remain board-man- 
dated and involve no explicit legislation. The remaining third are explicitly re- 
quired by legislation. 

While the inventory indicated no change in the dominance of this institution- cen- 
tered policy approach to assessment, results did suggest a number of changes in 
the ways in which states were carrying it out. Most apparent here was a growing 
de-emphasis on active enforcement, largely because of policy preoccupation with 
other matters. Though policies are in place, there was significant evidence that 
compliance is being de-emphasized and that assessment has become a "back- 
burner" activity in many states. Coupled with this was growing frustration with 
the institution-centered approach as a means to effectively address growing de- 
mands for public accountability. As a result, in about a third of the states falling in 
the "institution-centered" policy category, institutional reporting is being supple- 
mented by the development of common — but largely non-outcomes-based — indi- 
cators of performance. 
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Chart 1 

Status of State Assessment Initiatives in 1995 





Initiative 

Name 


Date 

Established 


Type of 
Initiative 


Common 

Instruments 


Public 

Reporting 


AL 


Assessment Policy 


1990 


I nsti tution-Centered 


No 


Voluntary 


AK 


(not available) 










AZ 


Regents Mandate 


1986 


1 nstitution-Centered 


No 


Annual 


AR 


Act 98 


1989 


Comprehensive 


Yes 


Annual 


CA 


HE Assessment Act 


1990 


1 nstitution-Centered 


No 


By System 


CO 


HB 1187 


1985 


1 nstitution-Centered 


No 


Annual 


CT 


Strategic Plan 


1989 


Institution-Centered 


No 


Voluntary 


DE 


(no initiative) 










FL 


CLAST 


1981 


Gatekeeping Test 


Yes 


Annual 


GA 


Planning Policy 


1989 


Institution-Centered 


Yes 


Annual 


HI 


Acts 371 


1989 


I nstitution-Centered 


No 


Annual 


ID 


Outcomes Policy 


1988 


Institution-Centered 


No 


Periodic 


IL 


Review of UG Educ. 


1986-90 


I nstitution-Centered 


No 


Periodic 


IN 


(no initiative) 










IA 


Program Review 


1991 


Institution-Centered 


No 


5-Yr. Cycle 


KS 


Assessment Policy 


1988 


Institution-Centered 


No 


Annual 


KY 


KAEP 


1990 


Institution-Centered 


No 


Annual 


LA 


(not available) 










ME 


Planning Goals 


1988/94 


Institution-Centered 


No 


Periodic 


MD 


HE Reorg. Act 


1988/91 


Institution-Centered 


No 


Annual 


MA 


(no initiative-"voIuntary" guidelines) 








MI 


(no initiative-no authority) 








MN 


(no initiative-no authority) 








MS 


(not available) 










MO 


Assessment Program 


1986/87 


Institution-Centered 


No 


Annual 


MT 


(no initiative) 










NE 


(Program Review) 










NV 


Assessment Policy 


1989 


Institution-Centered 


No 


Biennial 


NH 


Board Goals 


198 


Institution-Centered 


No 


Voluntary 


NJ 


(no initiative-"evolving situation") 








NM 


Report Card 


1990 


Institution-Centered 


No 


Annual 


NY 


Board of Ed Policy 


cl980 


Program Review 


No 


Cyclical 


NC 


Board Policy 


— 


Program Review 


No 


Cyclical 


ND 


Strategic Plan 


1996 


Accred itation-Oriented 


No 


Cyclical 



O 
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Chart 1 (Continued) 

Status of State Assessment Initiatives in 1995 





Initiative Date 

Name Established 


Type of Common 

Initiative Instruments 


Public 

Renortine 


OH 


SB140 


1989 


Institution-Centered 


No 


Annual 


OK 


Regents Policy 


1991 


Institution-Centered 


No 


Annual 


OR 


Board Policy 


1991 


Institution-Centered 


No 


Annual 


PA 


(no initiative-no authority) 








R1 


Board Policy 


1986 


Institution-Centered 


No 


Voluntary 


SC 


Cutting Edge/Act 255 


1988/92 


Institution-Centered 


No 


Annual 


SD 


Assessment Policy 


1996 


Comprehensive 


Yes 


Annual 


TN 


Performance Funding 


1979 


Comprehensive 


Yes 


Annual 


TX 


TASP 


1989 


Basic Skills/Gatekeeping 


Yes 


Annual 


UT 


Assessment Policy 


1992 


Institution-Centered 


No 


Biennial 


VT 


(no initiative) 










VA 


Assessment Program 


1986 


Institution-Centered 


No 


Biennial 


WA 


Assessment Policy 


1989 


Institution-Centered 


No 


Biennial 


WV 


Assessment Policy 


1989 


Institution-Centered 


No 


Periodic 


W1 


Accountability Policy 


1993 


Comprehensive 


Yes 


3-Yr. Cycle 


WY 


(no initiative-no authority) 








PR 


Assessment Policy 




Institution-Centered 


No 


Periodic 



For both reasons, moreover, a number of states are actively attempting to "stream- 
line" their approaches to assessment reporting. Virginia, for instance, is adopting a 
peer-based oral reporting and review process and Colorado is examining ways to 
simplify required written reports. 

Only about fifteen percent of the states reported that a common collegiate outcome 
measure of any kind was in place or under development. This set of states, in turn, 
reported using a varied set of instruments and approaches (see Chart 2). Florida, 
Texas and Georgia continue to maintain competency testing programs in at least 
one domain for all students in public institutions — programs that have been in 
place for more than a decade. Similarly well established is Tennessee’s "perform- 
ance-funding" approach, which tests samples of students in a range of general edu- 
cation domains as a part of a statewide incentive-allocation system. Newer 
programs follow these leads. Wisconsin periodically examines samples of students 
using national tests of general education — an approach also followed by South Da- 
kota in 1986-88, while Arkansas uses similar instruments but tests all students — 
an approach being adopted by South Dakota as well through a mandate just 
enacted by its Board of Regents after an eight-year hiatus in common testing. 



Though the instruments used in these seven states roughly address some of the do- 
mains of Goal 6.5, they are sufficiently varied in content that meaningful aggrega- 
tion or comparison as part of a national indicators approach is not feasible. 

A further four states included in the "institution-centered" policy category report 
that they are "actively considering" common testing in order to help address grow- 
ing demands for public accountability, while two more require that institutions use 
at least one nationally normed examination as part of their "institution-centered" 
approach. At the same time, survey results suggest that state interest in common 
measures remains strong. Seventeen states now collect and report comparative 



Chart 2 

Instruments Employed by States with Common Testing 



State 


Instrument 


Test-taking 

Population 


AR 


ACT-CAAP 


All Students 


FL 


CLASTAII Students 




GA 


Regents Writing Exam 


All Students 


SD 


ACT-COMP (1986-88) 


Institutional Samples 




ACT-CAAP (1996) 


All Students 


TN 


ACT/COMP/C-BASE 


Institutional Samples 


TX 


TASP 


All Students 


WI 


ACT-CAAP 


Institutional Samples 


In addition: 


AL, CO, VA, WV “considering” common testing 






indicators of institutional performance. In addition, according to the 1995 SHEEO 
survey, some thirty-five states now compile and report institutional gradu- 
ation/completion rates centrally using common definitions — a result that would 
have been unthinkable a decade ago (Russell and Chisholm 1995). Finally, in 
other parts of their background papers, states overwhelmingly cited "lack of appro- 
priate instruments" and "costs of implementation and development" as their pri- 
mary reasons for not moving forward on this agenda, while they mentioned far 
fewer substantive policy objections to this approach. The absence of common out- 
comes measures across states thus appears less a matter of deliberate policy 
choice than one of operational and fiscal necessity. 

2. A Changing Context for State Assessment Policy. In many ways more signifi- 
cant than the overall pattern of state activities in collegiate assessment was their 
changing character. The state background papers suggested strongly that the pol- 
icy premises for adopting assessment had shifted markedly since the topic’s emer- 
gence as a notable realm of state action in the mid-eighties. First, states indicated 
that assessment is becoming part of a much larger state-level policy picture. While 
observers of emerging state assessment initiatives in the mid-eighties charac- 
terized many as "trains on their own track" not visibly connected to wider net- 
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works of regulation and incentive (Ewell and Boyer 1988), most of the assessment 
policies reported in the 1995 inventory were widely linked to other initiatives. 

One dimension of this deficiency already mentioned is the continuing develop- 
ment of statewide performance indicators that include more than just outcomes. 
Confirming the results of a 1993 study conducted by the Education Commission 
of the States (Ruppert 1994), information obtained from some seventeen states 
suggested that assessment results are being incorporated into broader account- 
ability reports that also include information like completion rates, faculty work- 
loads, student experiences, and instructional costs. Also more apparent are visible 
linkages to funding. While in 1989 only one state program (Tennessee) tied an in- 
stitution’s performance on student achievement measures with additional funding, 
the 1995 inventory documented three more such cases (Missouri, Kentucky and 
Arkansas), with about ten further states operating addition-to-base or categorical 
funding programs with some connection to outcomes. 

More fundamentally, states reported that assessment activities are embedded in 
the architecture of a number of new or "re-structured" approaches to basic opera- 
tions. Probably the most prominent example here is new ways to address colle- 
giate admissions, with several states describing attempts to replace traditional 
college entrance examinations with competency-based approaches founded on 
portfolio review or performance-based assessment. The most visible such initia- 
tives are in Oregon, Kentucky, and Florida. Other states noted that actions such as 
these were under consideration and that ongoing K-12 reforms and the national 
"New Standards" project in elementary/secondary education is increasing the po- 
litical pressure on higher education to demonstrate its own effectiveness. Similar 
developments were documented in the realm of inter-institutional articulation in 
other states, where traditional course-by-course approaches were becoming in- 
creasingly burdensome. Finally, a number of western states — led by Utah and 
Colorado — are exploring the establishment of a technology-based "virtual univer- 
sity" to credential achievement, using authentic assessment approaches. All three 
types of initiatives build assessment activities directly into their foundations, but 
for operational not indicative purposes. 

The fact that such alternatives are being considered at all, moreover, is a product 
of significantly altered political conditions. As noted, most states suggested that 
assessment as a policy agenda had moved to a "back-burner" largely because of 
the need to attend to much more basic problems. The first of these is sheer politi- 
cal instability. Indeed, two of the agencies reporting on their assessment activities 
in 1989 had been abolished by 1995 — including one of the most prominent and 
comprehensive such programs (New Jersey). Several others noted that dispropor- 
tionate levels of agency attention were being devoted to matters of sheer bureau- 
cratic survival. A second such problem is fiscal. While state higher education 
budgets are generally better than two years ago, many states indicated that assess- 
ment could no longer be justified as an "add-on" expenditure intended solely to 
provide data, but instead had to be made part of a set of wider re-structuring initia- 
tives intended to improve systemic efficiency. In this regard, a number of states 
explicitly mentioned the fact that regional accreditors were beginning to assume 
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the major burden of stimulating "institution-centered” assessment activities, and 
that independent state action in this arena may no longer be needed. Finally, most 
agreed that accountability demands on higher education are much more urgent 
and sharply-focused than a decade ago and, as noted, that decentralized assess- 
ment approaches are inadequate to meet such demands. These sentiments ap- 
peared to confirm results of an ACT survey on the future demand for assessment 
whose results were also reported at the workshop (Steele and Lutz 1995): if cred- 
ible instruments and adequate funding were made available, states would probably 
have little hesitation in adopting broader assessment programs. In the short run, 
however, few states appeared confident that either precondition could be met. 

Reflecting this final point was considerable uncertainty among states about the de- 
gree to which the improvement of higher education quality itself remains a major 
public policy issue. As noted, this was the issue that originally spawned assess- 
ment in the early eighties, and the rhetoric of quality has been strongly linked to 
assessment ever since. Consistent with results of the NEA study of legislative lead- 
ers (Ruppert 1 996), however, state responses suggested that quality improvement 
in higher education was less a leading item on most state political agendas than it 
had been in the past. On the one hand, issues of health care, prisons, and tax re- 
form are crowding out issues of education generally. On the other, political lead- 
ers are increasingly seeing higher education as a "private good" that principally 
benefits individuals. As a result, they are far less willing to use scarce public 
funds to enhance it. Both developments, if true, have profound implications for as- 
sessment as an element of policy. The first suggests that assessment will only com- 
mand a political constituency if it is linked to more fundamental changes in the 
way higher education operates. Funding a program that seeks only to "bench- 
mark" achievement will attract little political support. The second suggests that 
the wider agenda.of standards-driven educational reform, of which the National 
Education Goals are a part, is itself in trouble. Neither contextual point suggests 
that there can be much immediate progress in building a national indicator of col- 
legiate attainment — whether from the federal level down, or from the state level 
up. 

B. Perceived Obstacles and Needs 

In addition to asking state respondents to describe their assessment activities, the back- 
ground papers requested that states describe what they perceived as important obstacles to 
assessment, some of their most commonly-encountered methodological problems, and what 
they saw as most needed in order to further a common assessment agenda. A total of twenty- 
two states provided information of this kind as part of their background papers. 

1 . Most Important Policy-Level Obstacles. States provided a total of 44 comments 

about the most important obstacles to assessment perceived or encountered. In de- 
scending order of frequency, these comments clustered around the following 
broad categories: 
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• high costs of development for assessment instruments (1 1 comments). 

• institutional resistance, especially from faculty and at research universities (1 1 
comments). 

• excessive diversity of assessment settings in postsecondary education — both 
with respect to mission differences among institutions and variations in the 
types of students served; the main theme here was that a single assessment ap- 
proach would probably not be appropriate across all settings (8 comments). 

• doubts about the continuing policy utility of assessment — especially in being 
able to communicate complex results to lay audiences in ways that would not 
be misinterpreted (6 comments). 

• lack of the requisite authority on the part of state agencies to undertake assess- 
ment in the first place (3 comments). 

These obstacles correspond closely to those identified by past inventories of state 
assessment activities, and to considerable anecdotal evidence as well. 

2. Most Important Methodological Problems. States provided a total of 39 com- 
ments to a question about the most important technical or methodological prob- 
lems that they have encountered or expect in the realm of collegiate assessment. 
Here the pattern of commentary clustered strongly around several themes: 

• lack of appropriate instruments; especially cited here were reliable "authentic" 
or performance-based instruments, and instruments to assess skills developed 
through collegiate general education programs such as critical thinking or effec- 
tive communication; frequently cited as well were "soft skills" such as interper- 
sonal skills or motivation (16 comments). 

• lack of faculty or community consensus about the actual domain to be assessed; 
again, skills and attitudes typically associated with undergraduate general educa- 
tion were prominently mentioned under this heading (9 comments). 

• a range of program implementation problems including lack of student motiva- 
tion to perform on non-required tests (3 comments), questions about data reli- 
ability and data use (5 comments), and concerns about lack of databases and 
state-level expertise in how to analyze assessment data (3 comments). 

The overwhelming emergence of instrumentation and domain consensus issues is 
not surprising, though considerable progress in the latter had already been made 
through the prior series of NCES Study Design Workshops, and ongoing work in 
achieving domain consensus on several key skills undertaken by the National Cen- 
ter for Postsecondary Teaching, Learning and Assessment (NCTLA) at Penn 
State. Clearly states are not aware of these results or do not believe they have yet 
yielded a workable common approach. 

3 . Most Important Needs. A total of 29 comments were generated about the most 
important actions needed to further a common assessment agenda. Again, state 
commentary tended to cluster quickly on this matter: 
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• additional funding; this comment, of course, was most often offered by those 
citing high costs as an obstacle (10 comments). 

• training and staff development; areas especially mentioned here included train- 
ing for state agency staff in how to develop and use assessment data, training 
for faculty and other institutional personnel in how to conduct sound local as- 
sessments, and networking among states themselves to share approaches and 
ideas (8 comments). 

• policy leadership from the federal government— especially in providing com- 
mon policy direction and definitions (7 responses), and in convincing state leg- 
islatures and executive offices that an ongoing assessment agenda is worth 
pursuing (4 comments). 

Taken together, this pattern of state commentary is consistent with results of the 
ACT survey that suggest a high potential state-level demand for assessment 
(Steele and Lutz 1 995). Most state papers cite primarily operational and logistical 
obstacles to proceeding further on this agenda. But results of the 1 995 inventory 
also make it clear that current state "interest" and "ability to act" in the realm of as- 
sessment are still far apart. 

III. Discussion Themes and Conclusions 

Plenary and breakout sessions at the workshop covered a broad array of topics and yielded a num- 
ber of salient points of agreement. A first set of discussion topics centered on the political context 
for assessment and emphasized the ways in which important premises at both the state and fed- 
eral levels have evolved since the NCES development process began some five years ago. A sec- 
ond set of topics concerned assessment methodology and, more particularly, the degree to which 
adequate approaches to postsecondary assessment have, can, and should be developed for use on 
a national basis. Finally, the two breakout sessions of the workshop established a number of 
points of consensus, as well as a range of conclusions about productive actions that might be 
taken. 

A. The Political Context 

An introduction to the workshop provided by Jean Griffith of NCES— as well as later ple- 
nary presentations by Charles Lenth of the Education Commission of the States, Mark Mu- 
sick of the Southern Regional Education Board, and Ken Nelson of the National Education 
Goals Panel outlined the national policy context within which any contemplated national 
postsecondary assessment would have to be crafted. The corresponding state perspective 
was provided by Peter Ewell of the National Center for Higher Education Management Sys- 
tems, and Joe Steele of the American College Testing Program, and "case studies" of spe- 
cific state efforts in Virginia, Missouri and Tennessee (Banta et. al. 1996). Open discussion 
following many of these presentations, as well as deliberations within the various breakout 
sessions, helped to clarify this political climate. 

All participants were conscious that the initially-contemplated federal role in instrument- de- 
velopment and data-col lection would likely be considerably constrained. Nevertheless, na- 
tional observers advanced a number of continuing reasons why an effort to assess 
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postsecondary outcomes on a national basis would still be needed. Dr. Griffith reasserted the 
premise of the workshop by noting that although the federal role appeared to be decreasing, 
"questions were still being asked" at all levels about postsecondary performance. As a re- 
sult, "finding ways for the Department of Education to help states assess student learning" 
remained a laudable goal. Mr. Nelson emphasized that the original conditions that led to the 
National Goals process still held and that the "standards movement [in K-12 education] is 
here to stay." But he also pointed out a growing political perception that "postsecondary edu- 
cation is lagging in this effort" by not becoming a visible part of increasingly-important 
standards-based reform movements at all levels. Sounding a familiar theme, he noted that in 
the absence of actions by higher education itself to address this perceived problem, policy- 
makers would not hesitate to apply "blunt instruments." This position was partially echoed 
by Dr. Musick, who observed that a slow but steady increase in interest about academic at- 
tainment was developing at all levels. Citing the fact that support for assessment is biparti- 
san and embraces both supporters and critics of higher education, he predicted that interest 
would "cycle back" to these topics in the long run. Recent statements of interest in estab- 
lishing "rising junior" examinations at the postsecondary level by governors in Arkansas, 
Kentucky and Colorado were cited as evidence of this continuing interest. 

State representatives acknowledged these concerns, but came from a different policy per- 
spective. As already noted in their background papers, most states reported that they were 
preoccupied with governance and fiscal difficulties, with higher education matters receiving 
considerably less attention than they had in the past. One result was that the "quality" issues 
driving the establishment of assessment initiatives in the mid- to late-eighties were now be- 
ing replaced with "productivity" issues that require much more straightforward data to track. 
More subtly, state assessment policies had generally not been substantially based on a 
"standards-driven" approach in the first place. Instead, like the regional accrediting agen- 
cies, these policies emphasized building local assessment capacity at the institutional level 
as an aid to instructional improvement. Indeed, a number of state representatives believed 
that state policies for higher education in all areas were increasingly moving away from 
"standards" and "mandates" to approaches based on "creating markets" and "managing in- 
centives." A key factor in ensuring the longevity of Tennessee’s performance funding ap- 
proach, for instance, was its use of positive incentives and the willingness of state officials 
to periodically re-examine standards and measures and revise them as needed (Banta et. al. 

1 996). Nevertheless, state representatives increasingly recognized the fact that decentralized 
assessment approaches, in Dr. Miller’s words, "don’t tell much of a story" to outside audi- 
ences. As a consequence, they felt growing pressures to come up with a summative measure 
of collegiate learning. As emphasized in their background papers, however, most felt that 
the task of creating and fielding such a measure was currently beyond the means of any sin- 
gle state. 

All parties agreed that for a number of operational reasons, increased general use of assess- 
ment techniques in postsecondary settings was likely. Dr. Musick reminded the workshop 
that questions about the effectiveness of remediation, of "bridge" programs among institu- 
tions or between high schools and colleges, and of increasingly-prominent technology- 
driven or distance-learning approaches were of rising policy concern, and that all would 
require information about the development of cognitive abilities. Dr. Lenth further empha- 
sized this point by asking participants to imagine a new kind of institution "located next to 
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the Information Superhighway" in which competency-based evaluation against clear per- 
formance benchmarks would be the norm. Using this vision, he emphasized that changes in 
technology and in the socio-political context for postsecondary delivery had moved the pol- 
icy conversation about assessment far beyond the agenda of "improving undergraduate edu- 
cation" that was typical of the 1980’s. 

Despite many differences in perspective, these discussions revealed a number of points of 
consensus. Echoing findings of the ACT survey (Steele and Lutz 1995), most agreed that 
suitable summative measures of student achievement would be useful to states in addressing 
a growing accountability problem for postsecondary education. But most also believed that 
appropriate measures were not currently available and that developing them was beyond the 
means of states acting individually. At the same time, unsettled fiscal conditions and uncer- 
tain political will remained major concerns at the state level. As a result, states appeared ba- 
sically willing to participate in developing some common approaches to measuring 
collegiate attainment, but only if such measures were consistent with local policy objectives 
of improving productivity and if the expected substantial costs of such an effort could be de- 
frayed by others. 

B. The Methodological Context 

Methodological perspectives were provided by presentations given by Elizabeth Jones 
(Jones and others 1995) of the National Center on Postsecondary Teaching Learning and As- 
sessment (NCTLA), Steven Dunbar of the University of Iowa, John Mazzeo of the National 
Assessment of Educational Progress (NAEP) program of the Educational Testing Service 
(ETS), and David Lutz of the American College Testing Program (ACT). Additional back- 
ground on the NAEP and National Adult Literacy Study (NALS) was provided by Gary 
Phillips and Andy Kolstad of the NCES staff. Once again, discussions following each of 
these plenary sessions provoked a range of additional comments from participants. 

From a purely technical standpoint, test designers saw no insurmountable obstacles to build- 
ing the kinds of assessments required. Experience with both the NAEP and with ACT pro- 
jects such as "Work Keys" sustain the conclusion that large-scale performance assessments 
covering key collegiate skills can be built and that, indeed, prototype measures addressing at 
least a part of the intended domain already exist in some form. Problems of more fully speci- 
fying this domain — and in such a way that the resulting measures remain credible across a 
diverse postsecondary community and with such key external audiences as employers and 
policymakers— proved more open to debate. Results of the>Delphi-based domain-specifica- 
tion project reported by Dr. Jones— as well as the outcomes of the two previous NCES 
Study Design Workshops — do suggest that such consensus is possible. But participants ex- 
perienced in the actual development of performance assessments at the state level empha- 
sized that even if "domain consensus" of this kind is achieved, it may be far too general. 
Instead, the necessary understanding and agreement must occur at the level of actual assess- 
ment items and tasks, and results must be communicated in a manner that concretely refer- 
ences actual performance standards and what students are specifically required to do to 
attain them. This point was reinforced by experience with the NAEP and NALS, where the 
initial establishment of performance levels and public reporting were consistently anchored 
in actual test materials. 



O 

ERIC 



14 20 



Deeper concerns about the actual uses to be made of such results, however, were widely 
voiced by workshop participants. Dr. Dunbar noted, for instance, that well-designed instru- 
ments were only one part of an effective assessment technology. Equally required is a sur- 
rounding "support structure" of policy and intended use that remains "stable enough” to 
give its results validity. Indeed emerging concepts of "validity" demand that the uses of as- 
sessment results, and the potential consequences of these uses, be taken fully into account. 
From the standpoint of instructional improvement, for example, Dr. Dunbar pointed out that 
the domains in question are not explicit but are rather "tacit" objectives of most postsecon- 
dary instruction; because few instructors would acknowledge that they are explicitly trying 
to develop such skills in college classrooms, attempts to use such results for improvement 
will be difficult. Other participants expressed substantial doubts about the real utility of such 
results at the state level beyond simple accountability, believing that local measures that 
could be linked more closely to actual curricular experiences at different institutions would 
be more helpful. Many similar comments emerged as conclusions to the workshop breakout 
sessions. 

Additional comments offered by workshop participants emphasized other technical features 
of the required assessment. One prominent theme was the need for contextual information 
about student population characteristics and the attributes of particular instructional settings. 
Many felt that contextual information of this kind was critical to drawing meaningful conclu- 
sions about institutional effectiveness, while for improvement purposes it was critical to link- 
ing obtained outcomes with specific policies, settings and experiences. Participants also 
believed that the most meaningful use of such measures was to show improvement over 
time — again emphasizing the need to obtain accurate and appropriate early benchmarks. In 
the same vein, most believed that longitudinal designs that could account for differences in 
entering student achievement levels, while also examining different paths of student devel- 
opment, would be beneficial. While sample-based assessment designs such as NAEP were 
viewed as the most viable approaches to achieving these objectives, some state-level repre- 
sentatives believed that testing all students for mastery was most appropriate — essentially 
holding individual students responsible for demonstrating achievement. Such comments re- 
flected previously-mentioned interest in using assessment devices like NAEP for opera- 
tional purposes — for instance in collegiate admissions. Finally, most agreed that extensive 
stakeholder involvement in assessment development was critical for success. As one partici- 
pant put it, a successful process of development should be neither "bottom-up" nor "top- 
down" in character. Instead it should involve active face-to-face dialogue in the design 
process among the various parties-at-interest. 

C. Results of Working Sessions 

The workshop’s two working sessions were intended to solicit feedback on a range of top- 
ics. A first set of working sessions examined the prospects for meaningfully aggregating ex- 
isting state-level assessments, as well as the feasibility (or desirability) of developing a 
NAEP-like measure of postsecondary attainment for use in common by the states. The sec- 
ond set of working sessions addressed specific steps that might be taken by both the states 
and NCES to further a common assessment agenda. 
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1 . Ability to Aggregate State Measures. Reports on the feasibility and utility of 
state-level common measures were submitted by each of five working groups. 
When asked whether existing state-level assessment activities can be used to gain 
insight into student learning from a national perspective, the clear consensus of 
opinion was that they are currently unable to do so. The reasons most commonly 
cited to support this conclusion were that few states use common assessment meas- 
ures for all institutions, and that even those that do so employ different instru- 
ments from state to state. Furthermore, the likelihood of obtaining sufficient 
commonality in the future was deemed low. Reasons cited were natural diversities 
across states in both policy approach and in the characteristics of their higher edu- 
cation systems. Participants in many groups also pointed out that current state- 
level assessment programs generally cover only those students who attend public 
colleges and universities, and that any national approach would need to include all 
students. Working groups disagreed, however, on the issue of how many states 
would be required to draw meaningful conclusions. Some concluded that all states 
would have to participate under common guidelines for results to be meaningful at 
the national level. Others felt that a useful national picture of collegiate attainment 
might be provided if up to a third of states could report outcomes on common di- 
mensions. Finally, one group noted that existing avenues for assessment like re- 
gional accreditation and state-level program review might be further capitalized 
upon; such activities occur regularly for most institutions and the opportunity 
might be taken to embed a small number of common assessment measures within 
them. 

2. Feasibility and Utility of a "NAEP-Like" Measure. When asked whether a na- 
tional survey of student learning similar to the NAEP or NALS could be used to 
meet state assessment needs, conclusions of the working groups were somewhat 
more optimistic — though conclusions still remained mixed. Two of five groups 
provided a qualified "yes" to this proposition, provided that agreement could be 
gained nationally on standards and definitions and that sufficient resources could 
be provided. The principal reason advanced was comparative: in the words of one, 
"it is important for states to have an idea of how they measure up nationally." One 
other group concluded that such an approach might be generally useful, but that 
states were sufficiently different from one another that they would need to de- 
velop their own assessment approaches anyway. The remaining two groups con- 
cluded that the results of such an assessment might be "interesting" and "useful as 
a benchmark" but that the implementation of such an assessment would not serve 
state interests because of widely differing state political conditions, the narrow- 
ness of the information that could be derived given the wide variety of outcomes 
that might be of interest to different states and institutions, and the likely resis- 
tance of institutions to the administration of such an assessment. 

Despite mixed responses, the working groups did have some specific suggestions 
about how a NAEP-like assessment at the college level might be designed and im- 
plemented. Rather than being administered at the end of four years, the suggestion 
was made that a sophomore-level test would be the most meaningful because it 
would occur at the end of most general education programs in college and could 
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be applied to students attending both two-year and four-year colleges. It was also 
widely believed that such an assessment, if developed, should be performance- 
based and should concentrate on "real-world skills." Because student motivation 
to complete such an assessment was seen as a major obstacle, one group felt that 
its content should be seen as "relevant" and "interesting" to students, while an- 
other group emphasized that individual feedback should be given to those taking 
the assessment. Finally, a suggestion was made that the development of such an 
assessment might be evolutionary — initially selecting as a pilot one subject area 
that already has a well- defined domain and associated indicators. 

3. What Is Needed at the State Level. The second set of working groups addressed 
questions about what might be done at the state level to further the assessment 
agenda, and what NCES might do to facilitate state-level efforts. Results of these 
working groups again highlighted some considerably different points of departure 
between state and federal actors. When asked about appropriate state-level action, 
most groups again began with the original premise grounding existing state assess- 
ment policies: to further institutional planning capacity and to meet growing ac- 
countability demands from legislators and the public. As a result, the principal 
objects of discussion centered on how to build assessment into already-existing 
state policy and quality assurance processes, and how to develop effective ways to 
induce institutions to participate more fully in assessment activities. Among the 
mechanisms most prominently mentioned under the first heading were statewide 
academic program review processes (for instance in Illinois or West Virginia). In 
a similar vein, many participants urged closer cooperation with regional accredita- 
tion — especially working with accreditors to develop more credible and common 
standards of performance. Turning to the second, participants emphasized the posi- 
tive role that state authorities could play in providing a forum for institutions to de- 
velop and share assessment approaches — a strategy illustrated by South Carolina’s 
state-sponsored but "institutionally-owned" statewide assessment network and 
similar initiatives in Virginia and Colorado. 

Not surprisingly given this point of departure, the primary suggestions for collec- 
tive action generated under this heading concerned improved networking and shar- 
ing "best practices." Virtually all the work groups identified as a high priority the 
establishment of a formal mechanism for the states to communicate about matters 
of common concern in assessment. In most cases, state-level participants agreed, 
current cross-state networking efforts sponsored by such agencies as SHEEO were 
not adequate because they did not involve the individuals at each agency who are 
directly responsible for coordinating assessment. Here information needs of two 
kinds were identified. First, practitioners wanted information on documentable 
"good practices." These concerned both the most effective data-gathering ap- 
proaches available and the most useful policies to induce institutions to adopt and 
use them. Participants felt that both kinds of information could be archived in data- 
base form for wider retrieval and analysis. Based on these documented cases, 
some participants believed, useful guidelines for the development of assessment 
practice under different conditions might be developed for use by all states. 

Equally required, others maintained, was a means for state-level assessment coor- 
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dinators to get directly in touch with one another to seek advice in as close to "real 
time" as possible. Quite a number, for instance, cited the NCES workshop itself as 
the first opportunity for peer exchange on the topic that they had experienced in 
some time. Possible mechanisms suggested to accomplish this objective were elec- 
tronic communications networks and periodic regional conferences. 

4. Actions that NCES Might Take. When asked the question of what NCES might 
do to help further state-level assessment activities, participants tended to think in 
many of the same terms. Consistent with the information provided in their back- 
ground papers, they identified funding as the most common single category of as- 
sistance. Interestingly, however, the use of funds to develop common instruments 
was not among their top priorities. Instead, initiatives aimed at identifying best 
practices and at facilitating communication among states again emerged as domi- 
nant. Although NCES was not felt to be the only actor that might play this role, 
opportunities for building such a network through the National Postsecondary 
Education Cooperative were recognized, and NCES was urged to examine these 
opportunities fully. Unique to NCES was the related role of providing national 
standards and definitions related to assessment. Within the framework of IPEDS, 
NCES was felt to be in an excellent position to begin to develop appropriate defi- 
nitions of key outcomes and standards of good practice in conducting state-based 
assessments, even though no common instruments for assessing collegiate achieve- 
ment could (or should) immediately be contemplated. The role of NCES in shap- 
ing state practice in graduation/completion-rate reporting — both through its draft 
graduation-rate reporting formats and its role in the rulemaking process for "Stu- 
dent Right-to-Know" — were favorably noted in this regard. Once such definitions 
and standards were in place, some felt, testing organizations and selected states 
would likely move forward to implement them. Consistent with standard-setting, 
moreover, some participants suggested that NCES might play a role in "certify- 
ing" or "auditing" state assessment practices, much as independent review teams 
examine industrial "quality assurance" processes through such mechanisms as the 
Malcolm Baldrige National Quality Awards or the international "ISO-9000" proc- 
ess. Working with regional accrediting bodies as an integral part of this process 
was again felt to be important. 

At the very least, some felt, national coordination might evolve a more common 
understanding across states of what actually constitutes "postsecondary assess- 
ment" and how the results of assessments should appropriately be communicated 
and used. As one participant cogently observed, even after ten years of practice 
states still have some very different ideas about the definition and appropriate con- 
tent of "state-level assessment," many of which legitimately extend far beyond a 
need to track progress on National Goal achievement. Among the broader pur- 
poses of this kind explicitly mentioned in working-group sessions were direct 
state-level accountability for resources invested, institutional program improve- 
ment, state-level priority-setting and policy development, and individual "con- 
sumer protection." 
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Pervading all comments at the breakout sessions, moreover, was the feeling that stronger na- 
tional leadership is neededfar more so than better technical expertise. Many state repre- 
sentatives appeared concerned about an erosion of impetus for assessment in their states. 
Consistent with their background papers, they believed that the federal government might 
help to supply such leadership — a process which had already begun through the National 
Goals effort. Participants made it clear that aligning state efforts to create a national picture 
of achievement would require an active "impetus for change" at both levels, as well as a set 
of well-articulated, compelling reasons for embarking on such an effort. Before discussing 
what or how to assess — or deciding on technical matters like instruments and sample sizes — 
they felt that all parties must be clear on exactly why the assessment is taking place. In 
short, although participants believed that the resulting data might be interesting and would 
not mind having such data if direct costs were minimal, the majority of states remained un- 
convinced that simply "filling a gap in national data collection" provided the necessary ra- 
tionale for developing an integrated national approach. 

IV. Conclusions and Next Steps 
A. Some Policy Implications 

Anyone involved with the evolution of state-level assessment in higher education over the 
last decade will discover many familiar themes in these Proceedings. Indeed, several work- 
shop participants confessed to a feeling of "deja vu" when discussing this topic, and noted 
that they had engaged in debates about most of its aspects many times before. Among the 
specific issues mentioned in this regard were the necessity of fully stating and achieving con- 
sensus on the purposes of assessment before embarking on the technology of instrument de- 
sign, the complexity of achieving agreement on the domain to be measured (as well as the 
wisdom of concentrating attention at the "item-level" in such discussions), and the difficulty 
of designing appropriate instruments that reflect authentic performances reliably but whose 
results can be summarized succinctly and understandably. 

Despite the perception of having “heard it all before,” workshop participants acknowledged 
that the circumstances under which these matters were being discussed had changed signifi- 
cantly. First, it was clear that immediate funding from either states or from federal sources 
to support a national assessment effort is unlikely to be available. As a result, any alterna- 
tives discussed would have to use existing vehicles and/or rely upon joint ventures and coop- 
erative efforts through which current resources might be leveraged. At the same time, 
substantial political uncertainties at both the state and the federal levels mean that it is 
equally unlikely that a clear policy direction will quickly emerge. This makes it all the more 
imperative that if something is to be done, that it be done through established organizational 
structures. 

Second, most states face pressing short-term demands to demonstrate the effectiveness of 
their higher education investments. As a result, many would welcome a straightforward way 
to document collegiate attainment using a common yardstick, were such an instrument avail- 
able. This stance, of course, represents a significant shift from the dominant policy position 
of a decade ago, when only a few states were willing to contemplate the use of common 
measures. 
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Third, assessment technology has progressed steadily during this period — substantiating the 
premise that credible, valid, and reliable performance-based approaches can in fact be devel- 
oped at the postsecondary level. Experience with the NAEP and NALS, state-level track-re- 
cords in developing comprehensive assessments like the New Jersey "Assessment of 
General Intellectual Skills (GIS)," and the growing sophistication of test-makers in the de- 
velopment of performance-based instruments at the K-12 level, all suggest that if sufficient 
funding and political capital were provided, and clear purposes for the effort established, ap- 
propriate instruments could in fact be built. 

Fourth and finally, significant changes are impending in the way business is done in postsec- 
ondary education that promise to render the use of assessment technologies more prominent 
and legitimate. Among the developments noted here were the use of competency-based ad- 
missions and articulation standards and the need to certify or evaluate learning obtained 
through alternative instructional-delivery mechanisms. As such developments progress, they 
will likely stimulate the use of more common standards and measures. 

Taken collectively, these substantial changes in context lead to a somewhat different set of 
policy answers to the "same old questions." From the technical standpoint, participants in 
the workshop agreed, a standard national approach to collegiate assessment can be crafted. 
The issues remaining are: why would states want such an instrument and for what? Essen- 
tially, results of the workshop suggested three possible policy answers, with quite different 
implications about the shape of the required assessment. 

• standard assessments of collegiate attainment are needed to meet increasingly press- 
ing state-level accountability demands. In the short term, the most immediate use for a 
summative assessment of collegiate skills is to demonstrate effectiveness on a state-by- 
state basis. Because such demands are generated within each state — and not all states are 
facing such demands — use of the same measure by every state would not be required. 

Such commonality might be desirable to benchmark state results, however. Given this 
purpose, a sample-based approach would be adequate because only aggregate reporting 
would be needed. 

• standard assessments of collegiate attainment are necessary to both drive and in- 
form state-level improvement efforts. Informing improvement and guiding instructional 
re-structuring and reform remain the primary reasons why states initially engaged in as- 
sessment — though they have done so on a decentralized basis. Nothing in this objective 
requires the use of similar measures across states and sample-based, aggregate-level meas- 
ures would be sufficient to meet it. Equally required in this case, however, would be two 
important additional conditions: data about contexts and instructional "good practices" 
that could be correlated with outcomes, and real linkages to policies and resources (both 
state and federal) that would induce states and institutions to act on the basis of what is 
found. 

• standard assessments of collegiate attainment will be increasingly needed in order to 
actually run the postsecondary enterprise. This policy answer reflects the growing pos- 
sibility that assessment technology will be directly embedded in state postsecondary op- 
erations in such areas as assuring college readiness, in certifying specific levels of 
attainment, or in regulating transferability among institutions. Unlike the other two policy 
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options, the technology suited to such applications would have to be applied to all applica- 
ble students, not just a sample. But construction of the required instruments might follow 
design principles similar to those used in NAEP. Operational-level results could then be 
aggregated for indicative purposes at the state and possibly national levels. A few existing 
state assessment programs — for instance the College Level Academic Skills Testing 
(CLAST) in Florida and the Texas Academic Skills Program (TASP) in Texas — already 
embody aspects of this "operational" application. 

Given these alternatives, and the current dearth of funds for instrument development, states 
would likely welcome a standard approach as a common benchmark for comparison if they 
could gain access to it at little cost and if it did not obstruct local policy objectives. Few, 
however, would adopt such an approach as a substitute for their current institution-focused 
programs or as a key component of state-level policy information. 

B. Some Possible Next Steps 

Results of the workshop indicate that there is no immediate possibility of using existing 
state-level measures to help construct a national picture of collegiate attainment. They fur- 
ther suggest that there are few prospects of states quickly developing such a capacity. But 
the workshop did yield a number of useful suggestions about how the states and NCES 
might work together more effectively to further a common assessment agenda. Among these 
potential next steps are four specific areas of potential cooperation. 

1 . Consortia of states — perhaps acting in partnership with NCES — might adopt 
an available instrument for use in common. Several states already use a stand- 
ardized instrument to assess collegiate general education. And despite doubts 
about the validity and utility of these measures, a growing number of states ex- 
press interest in fielding such an instrument purely to satisfy imperative demands 
for short-term accountability. As a result, support for using an available instru- 
ment in common might be fostered among a self-selected group of states — both to 
demonstrate effectiveness and to help benchmark their own institution-centered as- 
sessment efforts. Especially attractive in this regard might be regional consortia 
such as the Southern Regional Education Board (SREB) or the Western Interstate 
Commission on Higher Education (WICHE) where contextual differences among 
states might be minimized. Another alternative would be for states to collectively 
investigate a common problem or domain, using locally-chosen or individually-de- 
veloped instruments as appropriate. Examples here might include minority or ru- 
ral student achievement or, consistent with Goal 3, collegiate attainment in math 
and science. Under the former alternative, NCES might foster such collaboration 
through the National Postsecondary Education Cooperative (NPEC). Under the lat- 
ter, NCES and the states might actively identify and formulate a research topic for 
collective investigation, help develop an appropriate research design, and help co- 
ordinate and calibrate data-collection efforts across states. 

2. NCES might extend coverage of the National Assessment of Educational Pro- 
gress (NAEP) to a national sample of college students. Though this alternative 
would contribute little toward current state agendas, it has the substantial virtue of 
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using an existing vehicle for data-collection. While legitimate objections can be 
raised that an instrument such as NAEP is not capable of reflecting the full range 
and sophistication of collegiate domains, results of the NALS suggest that current 
college graduates have not "topped out" on far more basic skills, and that a na- 
tional benchmark would be useful. At the same time, administration of the NAEP 
to a sample of college students might help determine the feasibility of wider data- 
gathering at the state level — especially with regard to such intangibles as student 
motivation, faculty and institutional cooperation, and how the results of such an ef- 
fort might be interpreted and used. As above, the initiative could be focused in- 
itially on a single selected domain area. Results of the previous two NCES Study 
Design Workshops imply that assessing the domain of critical thinking/problem- 
solving is feasible. But more immediate progress might be made in areas like col- 
legiate-level communication or quantitative skills where a good deal of domain 
consensus and a body of widely-credible assessment technology already exist. 
Choosing the latter area would have the additional benefit of informing Goal 3. 

3. NCES, in partnership with other appropriate national bodies or associations, 
might embark upon a formal program of networking and information-shar- 
ing about assessment among state-level postsecondary assessment practitio- 
ners. The need for better lines of communication about current events and "best 
practices" in state-level assessment was apparent throughout the workshop. 
Mechanisms suggested included not only further conferences, but also use of the 
Internet and alternative forms of information exchange. Whatever the medium, all 
agreed that the focus should be placed upon substantive questions of policy rather 
than simply "assessment technology." Who should (or could) sponsor such a net- 
work, however, remains an open question. Organizations like AAHE and SHEEO 
are in many ways more logical "sponsors" for such networking than is NCES — 
and indeed are already partially engaged in such activities. But workshop partici- 
pants felt currently underserved by such organizations. Partnering with NCES 
through the establishment of a standing subcommittee on student outcomes as an 
integral part of the emerging NPEC might keep such organizations focused on 
serving those responsible for state assessment policy. Directly convening these in- 
dividuals on a regular basis through an arrangement similar to NCES’ annual Net- 
work Data Conference (or creating special session tracks within that conference) 
might also constitute viable options. 

4. NCES might attempt to proactively shape state assessment activities by evolv- 
ing a framework of standards, expectations, and "best practices" for state- 
level assessment. The primary precedent for this activity is NCES’ role in the 
development of standard practices for calculating and reporting postsecondary 
completion rates; despite the fact that no "standards" were eventually mandated in 
this arena, actual state practices have converged around definitions and procedures 
originally developed at the federal level. Analogous opportunities in the realm of 
state-level practice in collegiate assessment include the technical design of summa- 
tive assessments of collegiate achievement, specific guidelines on the domains 
and sample characteristics that would render individual state-level efforts both 
valid for local purposes and able to contribute to a national portrait of collegiate 
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attainment, and advice about effective practice in the public reporting of assess- 
ment results. Again, the emerging NPEC, in partnership with ECS and SHEEO, 
would provide a natural organizational framework for such an activity. 

While such steps are modest in comparison to construction of a national indicator of postsec- 
ondary attainment, they would arguably be as useful in fostering the development of effec- 
tive practice in this important area. And results of the workshop strongly suggest that 
assessment remains important to the states, despite changed conditions. Sustained grassroots 
efforts to improve its practice may therefore pay substantial long-term dividends, both to in- 
dividual states and to the nation as a whole. 
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APPENDIX A-l 



WORKSHOP AGENDA 



1995 NATIONAL CENTER FOR EDUCATION STATISTICS 
POSTSECONDARY EDUCATION ASSESSMENT PLANNING 

WORKSHOP 

December 7-8, 1995 
Key Bridge Marriott 
Arlington, VA 

THURSDAY, DECEMBER 7. 1995 



8:00-9:00: 


Breakfast Meeting of Workshop Team 


8:30-9.00: 


Continental Breakfast for all Participants 


9:00-9:05 


Opening Session (Introduction) - Sal Corrallo, NCES Project Director, 
National College Student Learning 


9:05-9:30: 


Keynote Speaker - Jeanne E. Griffith, Acting Commissioner, NCES - Purpose 
and Expectations - (Group) 


9:30-1 1 :30: 


Morning Session - “Current State Assessment Activities” 

Moderator: Robert Wallhaus, State Higher Education Executive Officers 


Presenters: 


Peter Ewell, NCHEMS - Summary of papers prepared by state participants 

Joe Steele, ACT - Research on Assessment Needs 

Trudy Banta, IUPUI - Performance Assessment in the States 


Questions and Comments 


11:45 -1:15: 


Luncheon - “National & State Assessment in the Current Political Climate” 
Moderator: Charles Lenth, Educational Commission of the States 


Speakers: 


Ken Nelson, The National Perspective 
Mark Musick, The State Perspective 


1:30-3:15: 


First Afternoon Seminar - “What Can Be Assessed at the State Level?” - 
Moderator: Pat Yaeger, NCPTLA 


Presenters: 


Beth Jones, NCPTLA - National Education Goals Skills To be Taught, Learned, 
and Assessed 

Peg Miller, - The Virginia Experience 

Ava Fajen, Richard Stein, and Charles Kupchella - Making Assessment 
Without Legislative Mandates: The Missouri Experience 


Questions and Comments 


3:15-3:30: 


Break 
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1995 NATIONAL CENTER FOR EDUCATION STATISTICS 
POSTSECONDARY EDUCATION ASSESSMENT PLANNING 





WORKSHOP 

December 7-8, 1995 
Key Bridge Marriott 
Arlington, VA 


3:30-5:30: 


Second Afternoon Seminar - "Methodological Considerations” 
Moderator: James Ratcliff, NCPTLA 


Presenters: 


Steve Dunbar, University of Iowa - An Overview of Issues and Concerns 
David Lutz, ACT - The ACT Assessment Approach 



John Mazzeo, ETS - The ETS Assessment Approach 

Questions and Comments 

7:30-9:30: Working Dinner in Hotel - Table groups to consider two questions. 

Leader/Recorder to be named. 

FRIDAY. DECEMBER 8. 1995 



8:00-8:30: 


Breakfast 


8:30-9:15: 


Brief notes from dinner table discussions. 


9:15-10:30: 


Small Group Sessions - Participants Assigned at Random to each group. 
Check board for room. Each session will consider three questions. 

Group 1 - Robert Wallhaus 
Group 2 - Charles Lenth 
Group 3 - James Ratcliff 
Group 4 - Patricia Yaeger 


10:30-10:45: 


Break 


10:45-11:30: 


Breakout Reports 


11:30-12:15: 


Open Mike 


12:15: 


Closing Comments - Peter Ewell, NCES 


12:30: 


Adjournment 


12:45-2:30: 


Working Lunch for Workshop Team and Invited Participants 
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APPENDIX A-3: BACKGROUND PAPER 
1995 NCES POSTSECONDARY EDUCATION ASSESSMENT 
PLANNING CONFERENCE 

Background: The National Center for Education (NCES) was reauthorized in 1994. Section 410, 
Cooperative Education Statistics Systems, states that: 

The Commissioner may establish one or more national cooperative education statistics 
systems for the purpose of producing and maintaining , with the cooperation of the states, 
comparable and useful information and data on elementary and secondary education, 
postsecondary education, and libraries, that are useful for policymakers at the federal, 
state, and local levels. In carrying out this section, the Commissioner may provide techni- 
cal assistance, and make grants and enter into cooperative agreements. 

The same act also included the Goals 2000 legislation which calls for improving the effectiveness 
and efficiency of learning at all levels of education. Of particular concern to NCES are Goal 3 
and Goal 6 (formerly Goal 5) as cited in the legislation. Both call for the enhance- ment of learn- 
ing of communication and thinking skills among students at the K-12 and college levels. NCES, 
as the primary agency concerned with the collection of data on the education industry, is ex- 
pected to provide a fair and accurate assessment of student achievement at all levels. The Na- 
tional Assessment of Educational Progress (NAEP) is cited in the legislation as the primary 
source of information for students in K-12. And although at the postsecondary level, the authoriz- 
ing legislation provides that NCES is to collect, analyze and disseminate statistics on student 
achievement at all levels of education. 

Collecting assessment information becausedt is authorized is but one reason for developing an as- 
sessment program. In light of major economic and social changes that have and are occurring 
both within the nation and the world community, students and parents alike consider a quality 
higher education a key component to obtaining and holding a job in the dynamic world today. 
Thus consumers are increasingly interested in obtaining information on the quality of the educa- 
tion level that schools and colleges must provide, not withstanding the increasing cost of obtain- 
ing a higher education. As a result state legislators and community leaders are increasingly called 
to assess higher education, which in turn has put increased pressure on state governing bodies for 
the assessment of student learning. These concerns had a large part in encouraging the develop- 
ment of the National Education Goals, formulated first by the nation’s governors and later 
adapted by Presidents Bush and Clinton. 

Current Activities: In direct response to the adoption of the National Education Goals, NCES 
has had an active program exploring ways to effectively assess the set of skills identified under 
Goal 6 Objective 5; higher order communication, problem solving, and critical thinking skills. 
Two planning workshops were conducted: the first, in 1991, to identify issues and concerns and 
second, in 1992, to identify the steps necessary to begin the process of skills identification. Both 
workshops provided a number of working papers. Collectively, participants in both workshops 
supported the notion that there are three sets of interrelated activities necessary for the achieve- 
ment of each objective and for fulfillment of an effective assessment process. These include: 
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1 . Identification of the Knowledge, Skills, and Abilities (KSA’s) Students Need to Know: 

There needs to be agreement as to what is to be taught and learned and then assessed from 
the perspective of faculty and staff, the larger community of citizens and the business sector. 
In particular, what knowledge, skills, and abilities are needed and at what level of achieve- 
ment? 

2. Enhancement of Teaching/Learning Functions (TLF): In designing an assessment, con- 
sideration needs to be given as to how the information collected through the assessment 
process might be used to enhance the teaching/learning of each of the skills. Further, the as- 
sessment instrument should accurately capture what students are being taught. 

3. The Implementation of the Assessment Process: Information collected as part of the as- 
sessment process may be used for accountability, fiduciary or legal requirements, program 
improvements, school and faculty performance, and funding decisions. However, the pur- 
pose of assessment must be well understood from the beginning of the design process. It is 
also at this point that consideration must be given to alternative approaches given academic, 
political, social and economic constraints that affect the assessment process. 

In 1993, NCES prepared a Request for Proposal (RFP) that focused upon skills identification for 
the college graduate. The purpose was to identify and obtain a consensus of the skills and levels 
of competency needed by college graduates beyond college. The results were to be used as a ba- 
sis for the development of a national survey of college student learning. A primary use of the in- 
formation was . to be used.on the status of efforts to meet the national education goals. The RFP 
was not funded due to budget constraints and it is unlikely that NCES will soon have sufficient 
resources to conduct a comprehensive national study, as originally envisaged. As a result, alterna- 
tive strategies were developed. 

The Penn State Projects: Over the 1993,1994, and 1995 fiscal years, NCES has funded three 
studies to identify the core skills needed by college graduates in the workplace. In FY 1993, the 
National Center for Postsecondary Teaching, Learning, and Assessment (NCPTLA) at Penn State 
conducted a two-stage Delphi, study to identify the writing, speaking and listening, and critical 
thinking skills needed by college graduates. Three panels, each with 200 members from the aca- 
demic and business communities, served as judges for the Delphi study. The final report was pub- 
lished in July 1995 1 . A similar study was funded in FY 1994 for problem solving, which is 
currently underway. A reading study was funded in FY 1995 and will begin in late 1995. This 
work will provide the initial listing of skills needed by college graduates as cited under the Goals 
2000 legislation. However, no further work is planned beyond these three projects. 

National Job Analysis Study: Over the same time period, NCES has been working with the De- 
partment of Labor to define a generic set of job skills in high performance workplace settings 
from a national perspective. This project was and is much more job specific. Using a national 
sample of workers, the first step was to identify sets of specific tasks and the needed skills based 
upon specific job experiences; This is in contrast to the' judgmental approach used in the 
NCPTLA project. Although the original intent was to identify the skills needed by high school 

1 Jones, E.A. and others. National Assessment of College Students Learning: Identifying College 
Graduates Essential Skills in Writing, Speech, Listening, and Critical Thinking. U.S. Department 
of Education, Office of Educational Research and Improvement, National Center for Education 
Statistics, Washington, D.C., July 1995 (NCES 95001a). 
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graduates, the study will provide a generic set of job skills. Once these have been identified, ap- 
propriate assessment instruments will be developed. At that time consideration will be given to 
the means of assessing the attainment of the; skills from a national perspective. The potential vehi- 
cle for a national assessment and the expected costs are unknown at this time. A final report is 
not expected to be available until at least the end of 1996. 

National Adult Literacy Study (NALS): A national assessment of adult literacy skills was com- 
pleted in 1994. NALS assessed the prose, document, and quantitative skills of adults of all ages 
and educational levels. NALS analyzed the variations in skills across major subgroups in the 
population as well as social and economic variables such as voting and earning. Although it 
could not provide a direct link between classroom learning and the skills needed in the work- 
force, it nonetheless provided a good overview of the various levels of adult literacy across the 
nation for a number of sub-groups. The design and size of the sample restricted the analysis to 
the nation, although a few states funded additional sampling to alloyv them to assess literacy 
within their state boundary. 

Problem: Given the shortage of resources and the need to obtain information on the state of 
learning across the nation, it is clear that alternative approaches to gathering information on the 
attainment of student learning at the college level need to be identified. A recent publication by 
the State Higher Education Executive Officers Association reports that all but two states have for- 
mal efforts focused upon the assessment of student learning. An ACT publication indicated that 
many of the skills areas noted in the National Education goals are being assessed, although un-. 
evenly. Information on assessment of college student learning collected by states offer a poten- 
tially valuable source of information across the nation. However, the approach to assessment and 
the way information is collected and analyzed differs across the states.. Can available state-level 
information provide an insight into the progress being made across the nation toward the achieve- 
ment of National Education Goal 6.5? To this end, it is important to note that the question is not 
whether the individual state level data can be summarized across states to provide a single na- 
tional summary report. Rather, can the individual state efforts be used to develop a collctive view 
of the progress being made to assure college graduates have the skills they need for admittance to 
and success in the workforce once they leave college? 

Planned Activity: A workshop is to be held, which will bring together directors of postsecon- 
dary assessment from each of the 50 states and selected territories. The meeting agenda will cen- 
ter around the following issues: 

1 . Identification of Skills Being Assessed. What knowledge or skills are being taught at the 
state level and how do they relate to the skills identified in the national education goals or 
other areas of national interest? What levels of achievement are expected? Are the skills be- 
ing assessed at the secondary level consistent with the skills being assessed at the postsecon- 
dary school level? 

2. Collection and Reporting of Skills. What is the frequency, reliability, consistency and com- 
patibility of the information collected across the states? How are these skills being assessed 
within institutions and across the state? At what grade level and how frequently? How are 
the results being used? 



3. Enhancement of Assessment at the State Level. What needs to be done to help states 
broaden and enhance their assessment programs to provide for complete coverage of skills 
in a more consistent and reliable manner? 

4. The Role of NCES. What is the appropriate role of NCES in assisting states and institutions 
gather and report the information and disseminate the results? 

As the ECS study indicated, most states, as well as many private institutions, are engaged in as- 
sessing student learning in one form or another. It would be valuable, both to policymaker and 
citizen alike, to gather this information and summarize the findings on a state by state basis. For 
reporting purposes, it is suggested that each state be thought of as a case study. The value of this 
approach is that individual states would be able to compare the results with comparable state ef- 
forts. This sharing of information can also be used to enhance the teaching/learning of these 
skills within a state. Again the focus of this discussion will be on the sharing of information on 
the development, implementation, and summary of assessment activities from a state perspective. 
To enhance this process, it will be necessary to use the same terminology and reporting format. 
Special consideration will be given to states which have successfully developed and implemented 
an assessment approach or program. 

Expected Outcomes: Two products are expected from this project. One is publication of an up- 
dated summary of current state postsecondary assessment activities and results prepared by work- 
shop participants from each state. The second will be a report which includes workshop 
proceedings on how state assessment activities might be enhanced along with a discussion on the 
potential use of assessment activities at the state level to support the national education goals. It 
will also include other papers and/or reports prepared for the working conference. 

Meeting Plans and Logistics: The meeting will begin with a morning session on the first day, 
December 7, and end at 12:30 the second day, December 8. Most sessions will be in a large 
group meeting although breakout rooms for 20 people each will be used the morning of the sec- 
ond day. It is expected that state representatives will attend along with a number of consultants 
and others from government and selected associations for a total of 90 participants. Transporta- 
tion and housing will be provided for the state representatives and consultants. State participants 
will be asked to prepare a summary of their state’s assessment activities. As noted, the workshop 
proceedings are to be published. A meeting agenda, prepared papers, and other information will 
be sent to participants prior to the meeting. 
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ALABAMA 



Origins of Assessment Interest in the State 

The state level assessment policy reported in 1990 is still in effect, although institutions no 
longer report annually on its implementation. This policy was initiated by the Alabama Commis- 
sion on Higher Education. The policy directs state higher education institutions to develop their 
own assessment procedures and to report the results in their respective Institutional Annual Plan- 
ning Statements. This reporting requirement has been discontinued. The policy, "Institutional Ef- 
fectiveness and Assessment of Student Learning Outcomes," was primarily for the purpose of 
statewide improvement of undergraduate education. At this point, the institutional assessment 
plans are voluntary and are developed by the institutions themselves. 

Funding 

No new or distinct funding was attached to the assessment policy. There are no designated state 
funds for assessment purposes, and each institution funds such activities through regular appro- 
priations. 

Statewide Articulation and General Studies Committee 

In 1994, the State Legislature passed Act 94-202, establishing an Articulation and General Stud- 
ies Committee under the auspices of the Alabama Commission on Higher Education. The Com- 
mittee is to develop a statewide general studies curriculum and articulation agreement by 1998, 
examine the need for a uniform course numbering system by 1999, and resolve problems in trans- 
ferring credit earned at one institution to another institution. The Committee has met monthly and 
is well involved in developing the statewide general studies curriculum. The completion of this 
project will enhance the efficiency of the higher education system by assuring students that they 
are taking appropriate coursework that will not have to be repeated. It will also provide a state- 
level framework for the resolution of articulation issues. This is an important step in relation to 
assessment of student attainment of knowledge and skills which are essential regardless of one’s 
major or vocational interests in that we can have greater assurance that assessment will relate to 
learning experiences common to students in all institutions. 

Exploration of the Need for External Standards of Educational Performance at Appropri- 
ate Points in the Undergraduate Curriculum 

During the 1 990-95 planning cycle, the Alabama Commission on Higher Education initiated dis- 
cussions to explore the possible development of a statewide rising junior examination. While con- 
sensus has not yet been reached on the need for the test, progress has been made, and this is a 
planning objective being recommended for continuation into the 1995-2000 planning period. In 
view of the work of the Articulation and General Studies Committee to develop a statewide com- 
mon core curriculum, a rising junior exam could provide an objective determination of student at- 
tainment of essential knowledge and skills prior to beginning upper division studies. 

Additionally, the staff of the Alabama Commission on Higher Education is looking at the need 
and feasibility of a statewide testing program for college seniors in their last term of study before 
graduation. 
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Institutional and Statewide Performance Indicators 

The 1995 report of the Alabama Joint Legislative Committee on Finances and Budgets stated. 
"The public is demanding accountability for the use of its money." The Alabama Commission on 
Higher Education is currently considering a recommendation that a state-level performance indi- 
cator system be put in place that will allow the state to gamer some sense of the trend and health 
of its higher education system and its progress toward achievement of long range goals. The rec- 
ommendation, if adopted, will promote the demonstration of quality by establishing a cluster of 
performance indicators that will be included in overall institutional and statewide assessment ef- 
forts. 
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ARKANSAS 



Entering Freshmen 

All full-time first-time freshmen must present scores on either the ACT, SAT, or ASSET. Ac- 
cordingly, institutions must place students in college-level classes or remedial programs. Stu- 
dents scoring below state standards in mathematics, reading, or composition must enroll in 
remedial work in the area or areas of weakness. 

Post-General Education Core (Rising Junior) 

All baccalaureate degree-seeking students must take the Arkansas Assessment of General Educa- 
tion (AAGE), which consists of a composition test and four CAAP tests (mathematics, science 
reasoning, writing, reading). After completing forty-five semester hours of coursework and be- 
fore completing sixty, students must receive a valid set of scores on CAAP. Since the AAGE is 
designed to provide data for the assessment of institutional performance, rather than performance 
of the individual student, no student who receives a valid score is prevented from advancing to 
the junior year. 

Performance funding goes to those institutions that demonstrate success in teaching courses in 
the thirty-five-hour State Minimum Core. CAAP covers the fields represented in the core: compo- 
sition, mathematics, natural sciences, arts and humanities, and social sciences. 

Exit 

Arkansas provides funding for demonstrated performance of students who have completed 
courses of study. The program is being phased in beginning with the fields of nursing and teacher 
education. 
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CALIFORNIA 



Origins/Interest/Authority for Postsecondary Student Assessment in California 

While California has no state-level mandate for assessment in 1990, the Legislature passed and 
the Governor signed the "Higher Education Assessment Act of 1990." The law states that the pri- 
mary purpose of assessment which the Act encourages but does not mandate — is to "improve 
teaching and learning as well as academic advising." Because California has not mandated any 
particular form of assessment that its public institutions must use, it has left assessment mecha- 
nisms to the discretion of the institutions. California’s Higher Education Assessment Act also de- 
clares the intent of the Legislature to monitor the performance of its public institutions in the 
following five areas: (1) diversification of the student body, (2) improved transfer rates, (3) im- 
proved retention rates, (4) diversification of faculty and staff, and (5) enhanced student learning 
and improvements in students’ knowledge, skills, and abilities. 

Primary Purpose of the Assessment Program 

While California does not have a state-mandated assessment program, each of its public institu- 
tions has some form of assessment activities. These activities focus primarily on entering stu- 
dents to determine the student’s level of English and math proficiency and to determine the 
course level at which the student most appropriately should be placed. 

Description of the Assessment Program 

University of California: Entering students are assessed to determine their writing proficiency. 

If they fail the English writing examination, they must enroll in Subject A (an introductory Eng- 
lish course). 

California State University: Entering students are assessed to determine if they possess profi- 
ciency in writing and mathematics. If they fail to score high enough on these examinations, they 
must enroll in introductory courses. These courses typically do not award degree credit. For Cali- 
fornia residents, results of those examinations are transmitted back to the high school from which 
the student graduated. In addition, at the State University, upper-division students are also as- 
sessed to determine that their writing skills are adequate for college graduation. A student cannot 
graduate from the State University until they have passed this upper-division writing test or 
earned a passing grade in an upper-division writing course. 

California Community Colleges: Through the community college "matriculation program," all 
new students who do not yet possess a college degree are required to take a series of tests to deter- 
mine their proficiency in reading, writing, and math. Based on the results of those tests, students 
are counseled as to which courses would be most suitable for them, however, such counseling is 
advisory only as students can enroll in any course offered by the community college. 

Validity of the Assessment Instrument and Reliability of the Process 

Unknown. 

Data and Test Results Collected Across the State 

Data from the above tests are not collected on a statewide basis. Each public institution can typi- 
cally provide information on the number of students who took and passed each exam and, in the 
case of the State University, report this information to the state’s high schools. 
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Reporting and Approval Requirements for Assessment Initiatives 

Institutions are not required to submit their assessment plans to the state for formal approval, al- 
though local community colleges must have their matriculation program assessment instruments 
approved by the state-level Community College Chancellor’s Office. 

Funding Available for Assessment Activities 

Funding for the assessment activities carried out by the state’s public institutions are not ear- 
marked specifically for that purpose, but rather are part of the state’s overall appropriation to the 
institution. As such, the state does not know how much is being spent on assessment activities. 

Findings from the Assessment Activities as They Relate to National Education Goal 6, Ob- 
jective 5 

California’s higher education assessment activities are not related to the National Education 
Goals and hence one cannot readily determine if Goal 6, Objective 5 is being achieved. 

Contact: Karl M. Engelbach 

Senior Fiscal and Policy Analyst 

California Postsecondary Education Commission 

1303 J Street, Suite 500 

Sacramento, California 95814 

Phone: (916)322-7331 

Fax: (916)327-4417 

E-mail:engelbac@cpec.ca.gov 
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COLORADO 



Origins/Interest/Authority of Postsecondary Student Assessment in Colorado 

In 1985, the Colorado legislature passed a bill that, among other things, required institutions to 
develop an accountability process to assess what students learned between entrance and gradu- 
ation. The statute, mindful of institutional role and mission, allowed each institution to develop 
its own plan based on the structure of the statute and the policies devised by the coordinating 
board. Accountability was part of a large bill that restructured the role of the coordinating board 
and, in many ways, was an afterthought primarily developed because of difficulties with the K- 
12 system. It did not address any identified deficiencies of the higher education system. 

Primary Purpose of the Assessment Program 

The purpose of accountability in Colorado was for higher education institutions to demonstrate 
public accountability and to increase the quality of undergraduate education. The legislation re- 
quired that institutions develop their plans (including goals, objectives, and measures) with maxi- 
mum public input and, later at the request of a prominent legislator, institutions were required to 
have the public review assessment results. The institutions report annually to the coordinating 
board (CCHE) and CCHE then develops a report for the Governor and the General Assembly. In- 
dividual institutions may develop shorter reports for public consumption, especially for legisla- 
tors in their districts. Some institutions have contributed information that resulted in newspaper 
articles. But in general the public accountability efforts in Colorado have had no impact. 

To meet the charge of producing well-educated undergraduates, the institutions have been more 
successful. Institutions developed goals, measures, and review results for all undergraduate de- 
grees and general education. Those results are reported annually to the coordinating board. One 
of the requirements of the coordinating board’s report is that institutions must discuss the 
changes made as a result of assessment findings. The institutions have made changes in curricu- 
lum, teaching skills and support services as a result of assessment. The primary value of this ac- 
tivity is the internal focus on undergraduate education to improve its quality. 

A Brief Description of the Assessment Program 

The coordinating board developed a policy that required institutions to assess students in the fol- 
lowing areas: 1) general education knowledge and skills, 2) skills and knowledge in the major/vo- 
cational area, 3) retention and graduation statistics, 4) alumni and student satisfaction, and 5) 
after-graduation performance. Institutions developed a set of academic program goals that were 
due in 1988. After discussions between coordinating board staff and institutional staff, goals 
were accepted or modified. The next step required institutions to build their plans for assessing 
students based on their approved goals. The plans were reviewed and evaluated by the National 
Center for Higher Education Management Systems and again a negotiation process between insti- 
tutions and the coordinating board occurred. Over a period of six months every institutional plan 
was approved. After an institution received approval, it began implementing its plan. The focus 
of all plans concomitant with the legislation is to assess graduates’ learning in programs; all plans 
measure learning in programs, not learning of individual students. Two-year institutions monitor 
the results of their basic skills programs, while all institutions measure skill attainment in the gen- 
eral education curricula. Community colleges monitor the skill and knowledge development of 
graduates with vocational degrees and certificates, as do the four year institutions for students ob- 
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taining baccalaureate degrees. There is no policy requirement to assess support services, but 
many institutions do so and report that data (e.g., library, tutoring, advising, etc.). The policy re- 
quires an annual institutional report based on the assessment results of their programs, and focus- 
ing on improvements made as a result of assessment. 

Means to Establish Validity of the Assessment Instrument and the Reliability of the Process 

Faculty have great flexibility to design their own goals and measures. Although the coordinating 
board may question some of the measures developed, in actuality there is little that it can do to 
make faculty change their measures. Many institutions have institutional research staff available 
to assist faculty in developing test questions, conduct surveys, and interpret data. It is ultimately 
the faculty, however, that choose the measures and decide if they are valid. 

Common Data or Test Results Collected Across the State 

At this time, there are no common test results (primarily because there are no common tests re- 
quired) available statewide. The coordinating board, with the support of key legislators and the 
Governor, is developing a proposal to collect certain “performance indicators” from each institu- 
tion to meet the need for public accountability. At this time, the process envisioned is all institu- 
tions will collect and report a common set of indicators in addition to a set of unique indicators 
chosen to reflect the institution’s role and mission, A certain unique set of indicators will be re- 
ported only at the state level. Only one indicator asks for the results of licensure and certification 
tests (although the Governor has suggested a test to be given to all graduates); and few address 
learning outcomes, other than student and employer satisfaction measures. 

Reporting or Approval Requirements of the Assessment Initiatives and by Whom 

The coordinating board has the ultimate authority for approval of the goals, plans, and annual re- 
ports submitted by institutions, but the governing boards are also required to approve these docu- 
ments before submittal to the coordinating board. Occasionally, a governing board will require 
that an institution rewrite a report. The majority of the interaction on assessment is between the 
institution and the coordinating board. 

Funding Available for the Past Three Fiscal Years 

The institutions must fund assessment through their base budget allocations; no additional mon- 
eys have ever been provided to implement assessment activities. Before the legislation was en- 
acted, a quality improvement program (in the early eighties) that allocated grants for improving 
instruction supported some early assessment efforts. Colorado’s coordinating board has the dubi- 
ous distinction of having the authority to decrease an institution’s budget by 2% if the institution 
does not implement an accountability program. Although the 2% provision provided leverage to 
the coordinating board at the beginning of the process, it could never be implemented politically. 

Brief Overview of Findings 

Since each institution develops its own program, there are no overall statewide findings. In gen- 
eral, the community colleges have not been successful in measuring the impact (if any) of basic 
skills courses. Instead some of their data shows that developmental courses may have no impact. 

General education skills of writing and computing are measured by all institutions with essays 
and math tests (usually). In general, institutions can show increases in writing skills from basic 
freshman classes; however, that increase may not be maintained throughout the student’s career 




54 50 



if the student is not required to continue to write in classes. Many faculty have commented that 
seniors do not write at the expected level. Math skills follow the same pattern. Some institutions 
develop oral presentation assessments using speech classes. Knowledge gain from general educa- 
tion classes is sparse and not well-documented. The majority of institutions have a distributed 
general education curriculum, and it is difficult to verify that the curriculum produced any spe- 
cific knowledge gain. Students will have taken a unique set of courses in a unique pattern of 
course taking. It is very difficult to say that choosing 20 courses randomly out of many possibili- 
ties will always produce the outcomes desired in vaguely-written general education goals. 

Faculty develop measures and review results of assessing the major/vocational skills with the 
most enthusiasm. Measures include papers, presentations, student surveys, capstone courses and 
tests. The majority of these measures test the skills and knowledge needed in the discipline/ voca- 
tional area, with few measures of the “softer skills” — better citizens, increased volunteerism, en- 
hanced aesthetic appreciation, etc. This is the area where most employers and faculty find 
students to be lacking the necessary skills. Employers agree that students (with the possible ex- 
ception of those holding certificates) have the necessary technical skills to perform the job, but 
do not have the "soft skills": communication skills, customer service, etc. 

Committees consisting of local employers and professionals working in the field review the re- 
sults and comment on the goals, measures, results, and possible changes. Committees provide a 
good review of technical skills, but they often don’t see measures or results on other skills. 

Special Assessment Needs 

Following from the answers in the previous questions, there are two areas where additional as- 
sessment is needed. The first need is "soft skills" and these should be defined by employers. Fac- 
ulty too often emphasize skills needed in their faculty positions (e.g., individual research papers 
and oral presentations as measures of communication skills) in their program measures that may 
be less relevant outside of academe. 

The second need is a methodology for assessing knowledge gain in general education to measure 
if general education goals are being met. A set of goals measuring general education, acceptable 
measures, and useful results are lacking at most of the institutions in Colorado. After dismissing 
standardized tests such as the ACT Profile and CAAP early in the assessment effort, institutions 
are beginning to use them again because institutions have not developed anything to replace them 
that is reliable and valid. It is not clear if standardized tests are valid for all general education cur- 
ricula, but the tests do produce reliable results that can be compared against national outcomes. 

Contact: Stephanie Cunningham 

Academic Officer 

Colorado Commission on Higher Education 

1300 Broadway, 2nd Floor 

Denver, CO 80203 

303-866-2723 

303-860-9750 (fax) 

steph@cche.state.co.us 
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GEORGIA 



Origins/Authority of Higher Education Student Assessment in the State 

Under the aegis of the Board of Regents’ Office, Office of Planning and Policy Analysis, an ad 
hoc Assessment Advisory Committee was appointed in 1987. Committee membership consisted 
of institutional presidents, vice presidents, institutional research and planning officers and system 
office staff. In December, 1989, the Board of Regents approved a "Planning and Assessment" 
policy statement that requires every institution in the University System to have a summary of 
significant assessment results and associated improvement objectives along with action plans by 
which improvements in effectiveness will be achieved. In addition, Board policy directs each in- 
stitution to describe the process by which systematic assessment of institutional effectiveness is 
conducted and the results of assessment are used to achieve institutional improvement. Policy 
also mandates that each institution will link its major budget allocations and other major aca- 
demic and administrative decisions to its assessment process. The policy also permits assessment 
procedures to differ from institution to institution, as long as each program includes the assess- 
ment of: basic academic skills at entry, general education, specific academic program areas, and 
all academic and administrative support programs. The formal Board policy is supplemented by a 
set of assessment "Resource Manuals," which provide structure for a standard assessment model, 
but provide great institutional flexibility in actual implementation. 

The Primary Purpose of the Assessment Program 

The primary purpose of the assessment program is improvement of the function assessed. Sum- 
maries of outcomes and expected results, descriptions of assessment methods, assessment results 
and improvement plans for each institution are maintained in the Board of Regents Office. It is 
the individual institution’s responsibility, and the responsibility of the faculty/staff of individual 
programs to adopt and implement their respective approaches to assessment. 

A Brief Description of the Assessment Program 

Underprepared entering students in the University System of Georgia are assessed through the 
Collegiate Placement Exam (CPE) or the COMPASS exam to determine their entering level of 
preparation for college work. The University System also conducts a Regents’ Test of reading 
and writing skills for rising juniors. 

General education is assessed according to predetermined learning outcomes and expected results 
in such areas as communications, mathematics and science, and cognitive reasoning. Each institu- 
tion establishes its own set of general education learning outcomes and assessment methods. 

The assessment of degree programs is the responsibility of each program’s faculty who define 
learning outcomes, select and implement assessment methods and analyze results and plan the in- 
dicated program improvement. 

All student learning outcomes are based on knowledge, skills, values, attitudes and/or behaviors 
that students are expected to acquire from the respective program. 

Non-academic outcomes, for all administrative and academic support functions, are defined by 
the staff who perform those functions. It also is their responsibility to select or design and exe- 
cute assessment methods and use the results to improve the effectiveness of the function assessed. 
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Validity of the Assessment Instrument and the Reliability of the Process 

Consistent with the decentralized "ownership" of the assessment process in the University Sys- 
tem, assessment methods are selected or developed and implemented by the faculty, as appropri- 
ate to the particular academic program being assessed. Validity and reliability are stressed in all 
System assessment literature, as are the use of multiple methods and the appropriateness of both 
qualitative and quantitative methods. The System Resource Manual on Assessment Methods of- 
fers examples and guidance in a wide variety of methodological approaches. 

The reliability of assessment methods is determined, ultimately, by the accuracy of the insight 
into program improvement that they provide. Program improvement will be evident in the greater 
effectiveness in student attainment of learning outcomes that is revealed by subsequent assess- 
ment cycles. 

Common, Statewide Data or Test Results 

Other than the CPE for some entering students and the Regents’ Test for rising juniors, the for- 
mer a placement test and the latter a graduation requirement, there are no common, statewide 
data or test results prescribed or collected in the University System. 

Reporting/Approval Requirements of the Assessment Process and by Whom Administered 

Institutions are required to submit summaries of their assessment processes, including a descrip- 
tion of the organizational structure and process and how it is administered; outcomes and descrip- 
tions of the assessment methods; examples of assessment data obtained by the process and 
improvements made as a result (preferably with evidence of improved effectiveness). Approval is 
not required, but the Board of Regents staff has responded to institutional submissions with de- 
tailed suggestions for assessment program improvement. Beginning in 1996, the critique of insti- 
tutional assessment summaries will be performed by peer review teams consisting of institutional 
personnel. 

Available Funding for the Last Three Years 

No special funding has been available to support outcomes assessment in the University System. 
Presidents in the system receive a lump sum allocation from the Chancellor, and they are ex- 
pected to fund assessment internally. There is no line item funding in Georgia. 

A Brief Overview of Findings 

Because the University System assessment program is still in the developmental stage, with im- 
plementation proceeding at each institution’s own pace, there is not a set of comparable findings 
available for any specific area. 

Special Assessment Needs in the State 

The University System approach to assessment is open-ended, institutionally based, and not 
linked directly with specific external goals. 
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Contact: 



General Education — Dr. Cathie Mayes Hudson 

Associate Vice Chancellor, Planning and Policy Analysis 
Major Area Programs — Dr. Joseph J. Szutz 

Assistant Vice Chancellor, Planning 

Board of Regents — University System of Georgia 

244 Washington Street, SW 

Atlanta, GA 30334 

(404) 656-2213 

(404) 657-4130 

chudson@mail.regents.peachnet.edu 

jszutz@mail.regents.peachnet.edu 



HAWAII 



Origin/Interest/Authority of Student Assessment in the Public Higher Education Sector 

In 1986, the State of Hawaii enacted Acts 320 and 321 granting the University of Hawaii addi- 
tional administrative authority in the areas of warrants, budgeting, disbursing, expenditures, and 
emergency bids. These measures were adopted for a three-year period and were commonly re- 
ferred to as the “flexibility legislation.” 

As part of Acts 320 and 321, the Legislative Auditor was charged “to assess and evaluate” any 
impact of the administrative changes set forth in the Acts “on the quality and effectiveness of in- 
struction, organized research, public service, academic support, student services, and institutional 
support program areas as applicable at the University....” Particular emphasis was to be given to 
the impact of general fund transfer authority upon student education. 

In 1987, the Legislature adopted resolutions requesting the University of Hawaii to develop and 
implement an educational assessment program. 

In 1989, Act 371 extended the administrative flexibility previously granted to the University of 
Hawaii and the Department of Education. This law required the University to submit an annual 
report to the Legislature on the progress of its educational assessment activities. This report was 
to include the status of educational assessment programs within the University and indicate inter- 
relationships between educational assessment activities conducted by the University and the De- 
partment of Education. 

Within the University, Board of Regents policy and executive policy directing the University’s 
educational assessment efforts were adopted in the 1989-90 time period. 

The Primary Purpose of Assessment Programs 

The purpose of the University of Hawaii’s assessment effort is provided in Executive Policy Sec- 
tion III.A., page 2 of 9. 

A Brief Description of the Assessment Program 

The University of Hawaii assessment program is described in its annual reports on this subject. 
Annual reports have been prepared since 1988. 

As can be seen from the reports, no one assessment instrument is used across the system. The en- 
tire assessment process is quite decentralized. 

Common Data or Test Results Collected Across the State 

Again, no common test results are collected state-wide. Samples of outcomes from the various as- 
sessment activities are summarized on pages 16-21 of the Board of Regents’ Briefing on Educa- 
tional Assessment, January 20, 1994. Also, the University system prepares an extensive array of 
annual information reports on students and courses, surveys entering and graduating students, 
and tracks student progress within and across the system. 
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Reporting or Approval Requirements of the Assessment Initiatives and by Whom 

Reporting requirements are summarized in the University’s Executive Policy, Section V.B.2. Spe- 
cifically, each campus prepares an annual summary report on assessment activities for the Office 
of the President. These reports are then compiled into a system report and shared with the Board 
of Regents and Legislature as appropriate. 

Funding Available for the Past Three Fiscal Years, by Program Area If Possible 

This information is not available. Assessment activities are not identifiable as separate budget 
lines at the campus or system level. The system office sets aside a small amount of money to sup- 
port specific assessment projects, but this would not adequately reflect the total support of assess- 
ment activities. 

A Brief Overview of Findings by Class Level 
This information is not available. 

Special Assessment Needs Identified in the State Over and Above Those Identified in the 
National Education Goals 

Hawaii has given special attention to assessing the interrelationships between the University of 
Hawaii and the Department of Education. 

Contact: Dr. Colleen O. Sathre 

Vice President for Planning and Policy 
University of Hawaii, 

2444 Dole Street, Bachman Hall 110 

Honolulu, HI 96822 

Phone: 808/956-7075 

FAX: 808/956-9119 

E-mai 1 : csathre@mvax.its.hawaii.edu 
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ILLINOIS 



Origins of Assessment Interest in the State 

The Illinois Board of Higher Education adopted policies designed to improve undergraduate edu- 
cation, particularly student achievement, scholarship, and general education in September 1986. 
The policies were updated in 1990 by the Board’s Committee on the Study of Undergraduate 
Education. 

Description of the Initiative 

The policies on undergraduate education called for colleges and universities to define their objec- 
tives for general education and skill development within baccalaureate and baccalaureate-transfer 
degree programs, to communicate these objectives to students, to assess individual student pro- 
gress toward meeting these objectives and to conduct regular reviews of the undergraduate educa- 
tion experience. Colleges and universities were to collect data on the progress, retention, and 
completion of all undergraduate students as a basis for regular review and improvement of the un- 
dergraduate curriculum and support services. The policies call for colleges and universities to use 
multiple measures and to assess student progress at appropriate intervals. 

At the state level, information systems were developed to support the Board’s policies by 
1) providing feedback to high schools on the achievement of their graduates at public universi- 
ties; 2) providing information to colleges and universities on the progress, persistence, and suc- 
cess of their students including students who transfer among community colleges and public 
universities; and 3) providing public universities with information about the occupational place- 
ment and success of their baccalaureate graduates. 

Individual institutions have undertaken a variety of assessment activities including writing exami- 
nations, team-graded common course final examinations, general education field examinations, 
and senior-year capstone seminars or other experiences. Many institutions restructured their gen- 
eral education requirements and redefined expectations for students’ learning prior to developing 
new assessment methods. 

Primary Purpose 

The purpose of the Board’s policies is the improvement of undergraduate education. Assessment 
supports this objective by improving the chances for success for individual students and by pro- 
viding information for examining the quality and effectiveness of programs and institutions. 

Common Data or Test Results 

Two information systems, the Shared Enrollment and Graduation System and the Baccalaureate 
Follow-up System, allow monitoring of statewide progress toward broad objectives for under- 
graduate education. The primary assessment activities are taking place on campuses where proc- 
esses and techniques are selected or developed. 

Reporting or Approval of Assessment Initiatives 

Public universities and community colleges are required to submit annually reports on reviews of 
undergraduate education. Assessment was the focus topic for reviews submitted in July 1993. 

The results of assessment activities are also referenced in program reviews and in institutions’ Pri- 
orities, Quality, and Productivity reports. 
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Funding 

No funding has been specifically targeted to assessment nor is funding linked to assessment re- 
sults. During the last decade, high priority has beeriplaced on undergraduate education in the 
budget development process and new resources have been allocated to this priority in most years. 

Contact: Kathleen K. Kelly 

Deputy Director, Academic Affairs 
Board of Higher Education 
4 West Old Capitol Plaza 
Springfield, Illinois 62701 
217-782-3442 
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INDIANA 



Origins/Interest/Authority for Postsecondary Student Assessment in Indiana 

Indiana has no state-level postsecondary student assessment program. Authority for postsecon- 
dary student learning remains vested in the postsecondary institutions. There is no legislative 
mandate for either state-level or institutionally based assessment. The Commission for Higher 
Education is a coordinating board with no authority, barring legislative action, for introducing 
postsecondary assessment. 

Some institutionally-specific assessment initiatives are underway, none of which are comprehen- 
sive. Ivy Tech State College uses ACT’s ASSET assessment for student intake and placement 
purposes. Vincennes University has introduced course-based performance assessment for some 
courses and programs. Ball State University has implemented a writing assessment for all stu- 
dents. Indiana University at Bloomington is exploring performance assessment related to general 
education. 

What the state does have is a set of experiences that may have the effect of raising interest in the 
postsecondary assessment of student learning. These include 

• Experience with K-12 assessment, and specifically with an attempt to shift the original Indiana 
Statewide Test of Educational Progress, or ISTEP toward a performance-based assessment pro- 
gram. 

In 1992 new legislation called for changes in K-12 testing, many of them based on the Oregon 
model. Proposed changes included a shift to performance-based assessment (I-PASS) at sev- 
eral grade levels including grade 12, and a high-stakes "Gateway" exam to be required for high 
school graduation. In anticipation of these new assessments at both the Gateway and grade 12 
levels, the public postsecondary institutions agreed to work with the State Board of Education 
on standard setting and on connecting the new standards with revised admissions procedures. 
The 1995 General Assembly revisited the assessment issue and decided to move forward more 
cautiously. In the near future, K-12 assessment will include both a norm-referenced, multiple 
choice component and a writing component. The initiative continues to evolve. At the postsec- 
ondary level, what remains is an interest, supported by the Commission, in encouraging institu- 
tions to pursue their own, independently designed, assessment initiatives. 

• A jointly adopted Board of Education/Commission for Higher Education plan for improving 
the fit between high school and college. 

In February, 1994, the two boards adopted a plan which, among other things, specified a high 
school core curriculum for all students. Implementing "Indiana Core 40" has required the de- 
velopment of course-specific competencies in core subject areas. Identifying these competen- 
cies has engaged college faculty along with high school teachers in team efforts. From this 
effort should follow university-based professional development workshops for high school 
teachers that will address performance assessment at the classroom or building level. 
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• A series of Department of Workforce Development initiatives related to tech prep and to the as- 
sessment of competencies acquired by students taking occupational programs. Much of this, es- 
pecially the latter effort, is based upon the Oregon model. 

• A state-level pilot project (Indiana Performance Assessment) organized by the Commission for 
Higher Education in 1991-92 and funded from non-state sources. 

The project developed about ten experimental language arts and mathematics assessments that 
were administered to high school and college students and also to adults seeking admission to 
Ivy Tech State College. The project resulted in a major assessment initiative within the Col- 
lege of Arts and Sciences at Indiana University in Bloomington. The project director also con- 
sulted with faculty at Indiana’s two public two-year institutions, although that has not yet led 
to any comprehensive effort to assess student learning at either institution. 

• A now-defunct effort to encourage individual institutions to develop measures for the assess- 
ment of student learning. Active through two biennia in the late 1980s, this effort was aban- 
doned in 1989, leaving assessment tacitly recognized as an institutional responsibility. 

Contact: Karen Rasmussen 

Commission for Higher Education 

101 West Ohio Street, Suite 550 

Indianapolis IN 46204 

Phone 317-464-4400/Fax 317-464-4410 

e-mail "karen@chevax.che.state.in.us"@chevax@mrgate 
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IOWA 



1 . Board of Regent policy adopted in 1991 requires every institution under the Board to under- 
take outcomes assessment for every program. This grew out of an interest in this office for 
improving student learning. 

2. Primary purpose is to improve student learning. Secondary purposes are to improve teach- 
ing and to provide an outcomes assessment component to on-going program review efforts. 

3. The choice of assessment varies from program to program. The specific choices as to what 
outcomes assessment will be undertaken is a faculty choice based on the nature of the aca- 
demic discipline involved. 

4. The validity is faculty determined, with assistance from the central administration. 

5. No common data or test results are collected across the state. Each program reports on its 
methods, procedures and use annually (although consideration is being given to a biennial 
collection). This report is provided in summary fashion to the Board. 

7. See above. 

8. Assessment activities are budgeted through each institution. 

9. It is not our purpose to assess “National Goals”. 

10. See above. 

Contact: Robert J. Barak 

Deputy Executive Director 
Board of Regents 
Old Historical Building 
E.12 & Grand Ave. 

Des Moines, IA 50319 
515-281-3934 Phone 
515-281-6420 FAX 
E-mail: rbarak@iastate.edu 
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KANSAS 



Origins of Assessment Interest in the State 

The staff and members of the Kansas Board of Regents have been the primary initiators of assess- 
ment in the state by developing and approving a format in October 1988. Institutions are to create 
their own assessment design and activities, however each design must assess basic skills, general 
education and student learning in the major. Since 1988, the Council of Chief Academic Officers 
and Council of Faculty Senate Presidents have led an effort to improve the assessment of the ma- 
jor. 

Description of the Initiative 

Based on the format approved by the Board of Regents, all public colleges and universities in the 
state have created their own assessment plan based on institutional role and mission. The institu- 
tional plans were approved by the Board of Regents in January 1989. The plans have been imple- 
mented over a three-year cycle. While segments of institutional assessments are being phased in, 
all plans are currently operational. 

According to the format, the plans had to create and identify expectations for baccalaureate de- 
gree students in three areas: basic skills, general education, and the major field of study. The 
plans had to further show how institutions will measure attainment of these expectations and use 
assessment data to improve programs. 

Primary Purpose 

The primary purposes of assessment are to demonstrate publicly the effectiveness of higher edu- 
cation and to stimulate appropriate educational reforms. Assessment results are reported to the 
Kansas Board of Regents. 

Are Common Data or Test Results Collected Across the State? 

Assessment is being conducted at the institutional level. There is no statewide mandate requiring 
uniform testing and data collection. 

Is Reporting or Approval Required of Assessment Initiatives? 

A schedule for reporting development and implementation of assessment activities was included 
in each institutional plan submitted to the Board of Regents. Institutions report on assessment of 
basic skills and general education annually; assessment of the major is reported once every three 
years. 

Funding 

Institutions are funding assessment activities with current appropriations. 

Comments 

Interest in assessment is expected to increase as progress reports from the institutions are submit- 
ted to the Board. Performance funding is also being studied as another initiative to strengthen col- 
leges and universities. 
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Contact: 



John F. Welsh, Ph.D. 

Director of Academic Affairs 
Kansas Board of Regents 
700 SW Harrison Suite 1410 
Topeka Kansas 66603-3760 
913-296-3421 - phone 
913-296-0983 -fax 
E-mail: John@KBOR.State.KS.US 




63 



KENTUCKY 



Origins of Assessment Interest in the State 

Legislation was passed in 1992 (codified as KRS 164.095) which specified accountability indica- 
tors to be measured through a collaborative effort between the Kentucky Council on Higher Edu- 
cation (KCHE) and the public institutions of higher education. In addition, an objective of the 
recently-approved Strategic Plan for Kentucky Higher Education 1996-2000 is for institutional 
programs to “identify and evaluate students’ educational outcomes relative to program priorities 
within institutional missions.” Finally, a system of performance funding is under development as 
part of an overall funding model for higher education. The framework for this system is based on 
the goals and objectives of the Strategic Plan. Thus, current assessment activities are based on 
the directives in the 1992 legislation and the planning/performance initiative. 

Prior to formal legislation, the 1990 General Assembly funded the Accountability Enhancement 
Program to be coordinated by the KCHE. This program awarded mini-grants to institutions for 
projects related to student outcomes assessment. 

Primary Purpose 

The accountability reporting is a response to increased public demand for evidence of effective- 
ness in higher education. The accountability reports are submitted to the governor and legislature, 
each December. In addition, reports are sent to KCHE, university governing board members, 
state agencies, faculty groups, non-profit education and economic development organizations, 
public libraries, and numerous individuals. Accountability results are used in updating and moni- 
toring implementation of the systemwide strategic plan, in the academic program review and ap- 
proval processes and in fiscal policy development. 

Institutions use accountability and other assessment results to (1) make program improvements, 
(2) measure institutional effectiveness, (3) improve student support services, (4) monitor student 
progress through studies of different populations, (5) improve public service programs, (6) en- 
hance strategic planning and funding decisions, and (7) meet program and institutional accredita- 
tion requirements. 

The Strategic Plan establishes system priorities. Performance funding is intended to recognize 
progress in key areas of the plan. 

Description of the Initiative 

The accountability legislation directed the KCHE and the institutions to develop and implement a 
systemwide accountability reporting process based on 14 specific indicators. The 14 indicators 
addressed in these reports respond to the following three questions: 

• How satisfied are... students, alumni, parents, clients, employers? 

• How successful are... education reform efforts, research and public service efforts, remedial ef- 
forts, students who take licensure exams? 
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• How many ... programs are accredited, students are enrolled, students persist and graduate, stu- 
dents transfer from community colleges, degrees are awarded, student credit hours are gener- 
ated, hours do faculty work, faculty hours are spent instructing and who teach the classes, 
rooms are used, course sections are added or overrides approved to meet student demand, cred- 
its/semesters to earn a degree? 

An accountability plan and detailed reporting procedures were developed in 1992-93. The 1993 
reports established baseline data for each institution on each indicator. Upon reviewing baseline 
data, institutional achievement goals for each indicator were approved by KCHE and reported in 
the 1994 reports. Progress will be assessed after a five-year period ending in 1997, although pro- 
gress toward achieving the goals is reported annually. All reporting is guided by a detailed Proce- 
dures Manual which establishes the definitions and reporting guidelines for each indicator. In 
most cases, results are provided by level of student or program, by discipline, by race and gender, 
and other appropriate breakdowns. 

Each year separate accountability reports are prepared for each institution and distributed to inter- 
ested parties throughout the Commonwealth. Institution-specific reports are developed to show 
progress of a given institution compared to itself and to offer the most useful format for policy- 
makers interested in the results. Individual reports are prepared for each of the eight universities, 
each of the 14 community colleges, and a summary for the community college system. An over- 
all system report presents aggregate data by type of institution and provides a system perspective 
of results over the reporting period. 

The Strategic Plan identifies selected desired results that would reflect achievement of system 
goals and objectives related to quality programs, educated citizenry, equal opportunities, eco- 
nomic development, quality of life, coordination and advocacy. In some cases, accountability re- 
porting will help measure results particularly related to educational attainment. The framework 
for the performance piece of the higher education funding model is based on the goals and objec- 
tives of the Strategic Plan. 

Common Data Collected Across the State 

Detailed accountability reporting procedures were developed for reporting consistency among 
the institutions. These procedures include definitions for all relevant data elements and common 
collection and reporting guidelines. 

Flexibility has been built into the reporting process in recognition of differences in institutional 
missions and planning objectives. Institutional reports measure each institution’s progress com- 
pared to its progress in previous years while using common data definitions across all institu- 
tions. A balance was also struck between commonality and individuality in the development of 
goals. Three types of goals were established against which to assess progress on the account- 
ability indicators: 

• Uniform Targets. These goals establish a minimum standard of acceptable performance for in- 
dicators for which continuous improvement over time would not automatically be expected 
(e.g., student scores on licensure exams). 
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• Institutional Targets. These goals set specific numerical targets when the direction of progress 
was the same across institutions and institutional gains could be measured easily using a stand- 
ard methodology (e.g., entry-level course pass rates for remediated students). 

• Evidence of Effective Strategic Planning. For indicators where institutions may differ in their 
assessment of what constitutes progress, guidelines require the institutions to place results in 
the context of their planning objectives, provide evidence of relevant activities and discuss in- 
stitutional progress toward meeting the objectives (e.g., enrollment data where institutions may 
strive to increase or to reduce their current enrollments based on their planning objectives). 

The Strategic Plan objective to identify and evaluate student educational outcomes allows each 
institution to determine the most appropriate assessment method(s) consistent with institutional 
missions which have been approved by KCHE. 

The performance funding measures make use of selected accountability indicators. Institutional 
missions drive the definition of several measures whereby different types of evidence are ac- 
cepted in the context of individual missions. 

Reporting or Approval Requirements 

The accountability plan and the 1997 goals were approved by the KCHE. Annual reporting of ac- 
countability results is required. Biennial status reports will be required on the results of efforts to 
meet the Strategic Plan objectives approved by KCHE. The performance funding system will re- 
quire approval by KCHE, along with routine reporting on results in preparation for making fund- 
ing allocations. 

Although KCHE approval is required in the areas of accountability, strategic planning and per- 
formance funding, all of these activities are developed in close coordination with the institutions. 
Work groups chaired by KCHE staff and composed of institutional representatives develop final 
draft materials for consideration by KCHE. 

Funding 

Each university and the community college system receives state funding to support account- 
ability reporting. The KCHE also receives an appropriation for administering the project. Ap- 
proximately $230,000 is appropriated to the Council and a total of $1.2 million is appropriated to 
the eight universities each year to support the implementation of the accountability program. 

A portion of the new money for the community colleges and universities will be allocated based 
on performance on specific measures effective with the 1996-98 appropriations. 

Overview of Findings 

The 1994 Kentucky system accountability report is attached. The 1995 report will be available in 
December. 

Comments 

Three interrelated initiatives support Kentucky higher education’s efforts to demonstrate its effec- 
tiveness: the annual accountability reporting process, the KCHE’s Strategic Plan for Kentucky 
Higher Education including a specific objective to evaluate students’ educational outcomes, and 
the revised Funding Model including a system of performance funding. The goals and objectives 
contained in the recently-revised Strategic Plan comprise the framework for selecting the per- 
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formance funding measures. The desired benefits resulting from the Strategic Plan and the per- 
formance funding measures consider but are not limited to the available accountability indicators. 

Contact: Joanne Lang 

Deputy Executive Director for Planning and Policy Studies 

Kentucky Council on Higher Education 

1024 Capital Center Drive, Suite 320 

Frankfort, KY 40601-8204 

Phone: 502/573-1555 

Fax: 502/573-1535 

E-mail: JLANG%CHE@MSMAIL. STATE. KY.US 
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MAINE 



Origins of Assessment Interest in the State 

The Chancellor of the University of Maine System was the primary initiator of assessment activi- 
ties in the State. Assessment was made a goal and a priority when Chancellor Robert L. Wood- 
bury took office in 1986. 

Shortly after that date, Chancellor Woodbury asked the Vice Chancellor for Academic Affairs to 
convene a Systemwide Assessment Committee to begin to focus institutional efforts on improv- 
ing teaching and learning. In a planning process involving study of assessment efforts in other 
states, a group of individuals representing all seven campuses in the system attended the AAHE 
Assessment Forum in Denver in 1987. Within the state, the assessment committee also con- 
ducted a conference on assessment with a broad range of attendance from all the campuses. The 
conference was held in November 1988 and was devoted entirely to student outcomes assess- 
ment. Out of these planning efforts, four major principles emerged to guide the development of 
assessment initiatives. The principles are: 

Assessment is much more than giving standardized tests. 

Each campus in the System has a unique mission so therefore the best assessment program for 
that campus will be one developed to reflect the mission. 

The most effective assessment is linked to teaching and learning in ways that will improve 
teaching and learning. 

The most effective assessment is supported and developed by faculty and viewed by faculty as 
a way to improve the learning process. 

The planning efforts in regard to assessment were underwritten in part by a pool of funds made 
available for the years 1988, 1989, and 1990 from the Vice Chancellor’s budget. The funds cov- 
ered the costs of four pilot assessment projects at the University of Maine at Farmington, a sum- 
mer project in computer science at the University of Maine, and planning efforts at the 
University of Maine at Fort Kent. 

With the completion of the planning phase, the campuses have begun to implement their individ- 
ual assessment procedures. While each institution has created its own process or processes, they 
share elements in common to assessment plans that are operational elsewhere in the country. The 
activities that the Maine campuses have chosen include student portfolios, capstone examina- 
tions in the major, surveys of students and alumni, focus groups, and the development of a Learn- 
ing Center on one campus as a focus for assessment efforts. Indeed, the assessment activities are 
program specific within a campus. 

Primary Purpose 

The primary purpose of the assessment initiative in Maine is the statewide improvement of un- 
dergraduate education. 

Description of Assessment Program 

There is no statewide assessment program. Assessment activities in Maine are conducted by the 
seven institutions. Examples are: 
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University of Maine. Honors Program capstone experience of researching and writing a senior 
thesis or project. 

University of Maine at Farmington. Teacher education programs engaged in developing per- 
formance based assessment. (“What does a beginning teacher need to know and be able to 
do?”) 

University of Maine at Machias. Behavioral Science students participate in an exit interview 
just prior to completion of the program. 

University of Southern Maine. Counselor education program surveyed employers of graduates 
to ascertain how well prepared employers feel these individuals are for their positions. 

Reporting or Approval of Assessment Initiatives 

Assessment has been incorporated into the planning goals for the University of Maine System 
which are approved by the Board of Trustees. Assessment activities are reported regularly to the 
Board. 

Funding 

The seven institutions of the University of Maine System receive a general annual appropriation 
from which they are expected to fund their assessment activities. 

Contact: Nancy MacKnight 

Vice Chancellor for Academic Affairs 

University of Maine System 

107 Maine Avenue 

Bangor, Maine 04401 

Phone 207-973-3232 

FAX 207-973-3296 

E-mail: nancym@maine.maine.edu 
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MARYLAND 



Origins of Student Assessment in the State 

The Higher Education Reorganization Act, passed by the Maryland General Assembly in 1988, 
established an accountability process for public colleges and universities in the state. According 
to the statute, each public institution must submit, and the Maryland Higher Education Commis- 
sion must review and approve, a student assessment plan (§11-304) and annual progress reports 
(§11-306). 

Primary Purpose of the Assessment Program 

The goal of the assessment program is to encourage public colleges and universities, and particu- 
larly their faculty, to improve student learning, instructional effectiveness, and curriculum. 

Description of the Assessment Program 

In 1991, the Maryland Higher Education Commission required each two- and four-year public 
campus to develop a plan for the assessment of undergraduate student learning outcomes. The 
Commission approved these plans and has received progress reports from the campuses for 
1992, 1993 and 1994. Each report 1) provided data on common and institution-specific indica- 
tors of student performance, 2) analyzed the significance of the data to student learning out- 
comes, and 3) discussed the implications of the assessment program for innovations and changes 
at the campus. 

Means Used to Establish the Reliability of the Process 

The Commission has issued reporting guidelines annually for the student assessment reports. To 
enhance reliability, the guidelines have remained relatively consistent each year. However, slight 
revisions have been made to eliminate indicators, such as trends in grade point averages, which 
were found to be of limited use in assessing student learning outcomes. 

Common Data or Test Results Collected 

The reporting guidelines require a discussion of eight common indicators of student learning: 

1 . Effectiveness of general education programs 

2. Student retention and graduation rates for all campuses and transfer patterns for community 
colleges 

3. Student evaluation of teaching 

4. Admission of undergraduates to post baccalaureate study 

5. Academic performance of community college students who transfer to a baccalaureate pro- 
gram 

6. Student performance on licensing, certification, and graduate admission examinations 

7. Employment rates of graduates 

8. Perceptions of graduates about the quality of their educational experience, as revealed in fol- 
low-up surveys 
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Campuses are asked to examine trend data based on these and optional institution-specific indica- 
tors and to explain the significance of their findings to the enhancement of student learning out- 
comes, particularly in relation to the mission of the institution. The campuses also are asked to 
discuss the impact of the findings on institutional policies, services and educational practices re- 
lated to student learning outcomes. This can include areas such as course content and prereq- 
uisites, teaching methods, entrance requirements, and student services, as well as the extent to 
which the assessment effort has had specific educational benefits for students. 

Reporting Requirements 

The Commission staff reviews these reports to determine whether assessment findings resulted 
in actions designed to enhance the instructional process. The Commission has reported to the 
General Assembly on the progress which institutions are making towards the improvement of 
student learning outcomes. This year, letters of appreciation and congratulation were sent by the 
Commission’s chief executive officer to those institutions which demonstrated the impact of as- 
sessment on teaching and learning. 

Funding Available 

The accountability process was part of a reorganization effort that included significant base fund- 
ing increases for higher education. These increases are expected to cover the costs of developing 
and implementing the assessment plan and progress reports. 

Overview of Findings 

It is evident from the reports that the assessment process is enhancing teaching and learning. Sev- 
eral institutions indicated that their assessment activities have resulted in the adoption of new 
policies and procedures to improve learning outcomes for all students and have had an impact on 
resource allocation. The reports indicate that most of the campuses are actively engaged in the 
following: 

• Reviewing general education courses. Several campuses have instituted strategies to 
strengthen the core curriculum. 

• Examining programs and curricula. Program reviews have resulted in enhancements to pro- 
grams in several instances. 

• Sharing assessment findings with the college community. Most institutions have established 
committees to oversee and strengthen assessment efforts. This approach has resulted in wider 
dissemination of data to support assessment. 

• Reviewing student support services. Many institutions are implementing policy and procedure 
changes as a result of student learning outcomes assessment. 

• Reviewing policies on placement testing. Community colleges seem to place more emphasis 
on placement testing as it relates to student learning outcomes assessment. Several colleges re- 
ported that placement policies and procedures need to be revised in order to insure that stu- 
dents requiring developmental education are served. 
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The document provides summaries of the student learning assessment activities of each public 
campus in the state along with a staff analysis. 

Other Comments 

The Commission’s chief executive officer appointed a work group representing all segments of 
higher education in the state and including faculty to develop performance accountability stand- 
ards to measure campus effectiveness. This action resulted from legislative interest in the crea- 
tion of specific accountability benchmarks and indicators that can help to measure how 
efficiently public funds invested in higher education are being used. The work group will issue 
its report this fall, and its efforts may bring changes in the accountability process in the state, in- 
cluding the way in which student learning outcomes assessment is conducted. 

Contact: Michael J. Keller 

Director of Policy Analysis and Research 
Maryland Higher Education Commission 
16 Francis Street 
Annapolis, MD 21401-1781 
PHONE: (410)974-2971 
FAX: (410) 974-5994 

E-MAIL: MHECPOIMKELLER@MDHEC.ATTMAIL.COM 
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MINNESOTA 



Origins/Authority 

The state legislature, with leadership of the Higher Education Coordinating Board, was the chief 
initiator of assessment activities in the state with the passage in 1987 of a bill establishing a Task 
Force on Post-Secondary Quality Assessment. The task force was directed to study the objec- 
tives of assessment and how it can be used to improve postsecondary education. The task force 
was also charged with establishing a pilot assessment program within each of the public postsec- 
ondary systems in the state. 

In addition to funding the six pilot programs, the 1989 legislature re-authorized the task force 
through June 1991 (it was scheduled to sunset in June 1989). The six pilot projects were funded 
on a two-year cycle which continued through 1991. 

Since 1991, there has been no statewide/state directed or mandated assessment of student learn- 
ing in Minnesota. In that year, sweeping governance changes were enacted requiring the techni- 
cal college, community college, and state university systems to be merged into one Minnesota 
State College and University System by July 1995. Beginning in 1993, the Higher Education Co- 
ordinating Board came under attack in the legislature and in the 1995 legislative session, the 
Higher Education Coordinating Board was abolished. A smaller Higher Education Services Of- 
fice (HESO) was created in its place, primarily to distribute financial aid, collect student enroll- 
ment, financial aid, and other data, regulate private schools and colleges, lead the state 
telecommunications initiative, and administer statewide and federal programs. Other activities, 
including student assessment were left to the individual systems. 

Minnesota policy-makers have not viewed establishing student assessment at the state level as a 
priority initiative. In addition to governance issues, policy-makers have been more concerned 
with postsecondary financing policies, providing remedial education for recent high school 
graduates, time to completion, and establishing system performance measures. Among the per- 
formance measures legislated in 1995 were retention, transfer and graduation rates; job place- 
ment; minority enrollment and retention; and increased credits generated by telecommunicated 
courses. Measures of student learning were not included for either the Minnesota State College 
and University System or the University of Minnesota. 

Contact: Dr. Leslie K. Mercer 

Director, Data & Programs 

Minnesota Higher Education Services Office 

400 Capitol Square Building 

550 Cedar Street 

St. Paul, MN 55101 

(612) 296-6869 (phone) 

(612) 297-8880 (fax) 

E-mail: mercer@heso. state. mn. us 
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MISSOURI 



Origins of Assessment Interest in the State 

By the mid-1980s Missouri was actively involved in discussions about state-level perspectives 
on assessment, and Northeast Missouri State University was establishing a national reputation 
for its value-added assessment program. In 1986, with strong backing from the governor, state 
educational leaders challenged all public institutions to establish assessment programs which 
would improve student academic performance. 

Description of the Initiative 

Missouri’s assessment initiative, while mandating assessment at the state level, was not legis- 
lated. Consistent with its tradition of local autonomy, Missouri has used a permissive approach 
that requires institutions to develop programs tailored to institutional missions, but does not man- 
date specific instruments or measurements. Beginning in 1987, institutions provided annual re- 
ports to the Coordinating Board about campus-based assessment programs. 

As demands for accountability increased, new goals for higher education, including the reporting 
of student outcomes data, were adopted by the Coordinating Board and agreed to by each cam- 
pus. Missouri also expanded the data collection efforts of its Student Achievement Study, which 
tracks student progress in the public sector from high school through college graduation. Data on 
performance indicators are collected on a regular basis and shared with all institutions, and with 
the Coordinating Board; this information has replaced the annual reports about assessment activi- 
ties. 

In early 1991, the assessment coordinators at the public four-year institutions began a grass-roots 
organization, the Missouri Assessment Consortium (MAC). This group, which has established a 
strong network for communication, has chosen to remain informal. Principles for assessment, to 
be used by each institution were adopted and promulgated by MAC. 

In searching for new ways to directly influence campus behavior, Missouri has also engaged in a 
performance funding initiative, Funding for Results (FFR), which seeks to link funding policies 
with assessment and accountability initiatives. A subset of the performance indicators, selected 
and refined with input from the institutions, has since 1991 been used as the basis for a perform- 
ance funding component in the proposed budgets of Missouri’s public institutions of higher edu- 
cation. 

With help from a grant awarded to the Coordinating Board by the Fund for the Improvement of 
Postsecondary Education (FIPSE) in 1994, Missouri is refining and expanding its performance 
funding program to emphasize the importance of teaching/learning issues. Missouri’s perform- 
ance funding program rewards institutions at two levels. The first level addresses the contribu- 
tion of institutions to the achievement of statewide goals, e.g., enhanced performance in general 
education and in the major. A second level involves rewards for designing and implementing 
mission-specific campus-level performance funding that is data-driven and emphasizes improve- 
ments in teaching and learning. Institutions are working on a variety of projects that require sys- 
tematic assessment of student performance. 





Primary Purpose 

The primary purpose of Missouri’s public policies on assessment is the statewide improvement 
of undergraduate education. A small portion of Missouri’s assessment initiatives involve gradu- 
ate education. In addition, Missouri intends to use assessment activities to demonstrate respon- 
siveness to demands for greater accountability. 

Are Common Data or Test Results Collected Across the State? 

Data on the broad set of performance indicators referenced above are collected annually. These 
include productivity, efficiency, and quality measures. Each campus also gathers data in support 
of campus-based assessment programs. 

Is Reporting or Approval Required of Assessment Initiatives? 

Institutions report their data on all state-level performance indicators annually, but the selection 
of specific assessment instruments is still based on institutional priorities. The individual per- 
formance funding models of the institutions are evaluated by a peer group process using speci- 
fied criteria, and subject to Coordinating Board approval. Institutions will be required to submit 
annual reports including evaluative data to the Coordinating Board concerning progress on imple- 
menting a campus-based performance funding program. 

Funding 

While rewards for doing assessment were included in the initial performance funding program es- 
tablished in Missouri, each year the amount of money recommended for simply doing assess- 
ment has been reduced. In general, however, institutions have been expected to fund assessment 
efforts from a reallocation of current resources and appropriations. Currently, discussions are tak- 
ing place concerning the costs of doing assessment and the extent to which performance funding 
should continue to reward institutions for engaging in assessment activities. 

As Missouri’s performance funding initiative matures, there has been a gradual shift to incorpo- 
rate quality markers that can be attached to more traditional productivity measures. Overall, the 
portion of Missouri’s higher education budget recommendation based on performance funding is 
set at the margins. In FY 1993, 1 percent of the core operating budget was recommended. Actual 
funding levels for FY 1996 were .9 percent for public two-year institutions and 1.2 percent for 
public four-year institutions. Since this money is built into the core operating budget, institutions 
have complete flexibility as to how they spend this money. For FY 1997, performance funding 
recommendations represented 2.3 percent of the public four-year core operating budget and 2.9 
percent of the public two-year core operating budget. These recommendations are under review 
by the Governor and the General Assembly. 

Comments 

Institutions are currently implementing pilot performance funding projects, and developing plans 
for fully-developed programs to be in place within the next year. These Funding for Results pro- 
grams are driven by systematic assessment. 
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Contact: 



Robert Stein 
Senior Associate 

Missouri Coordinating Board for Higher Education 
3515 Amazonas 
Jefferson City, MO 65109 
314-751-2361 



MONTANA 



Origins/Interest/Authority of Postsecondary Student Assessment in the State 

The postsecondary institutions in Montana are developing assessment programs in response to 
the standards established by the Northwest Association of Schools and Colleges, and each col- 
lege and university is developing a program consistent with its own mission. Montana has no 
mandated statewide postsecondary assessment initiative. The Commissioner’s Office will prob- 
ably collect various indicators of academic progress for internal use and to prepare presentations 
for state legislators. 

Primary Purpose of the Assessment Program 

Montana has no statewide assessment program. 

Brief Description of the Assessment Program 

Montana has no statewide assessment program. 

Means Used to Establish the Validity of the Assessment Instrument 
Montana does not have a common instrument used throughout the state. 

Common Data or Test Results Collected Across the State 

At present, enrollment reports in a common format are the only data collected by the state office 
from all institutions. 

Reporting or Approval Requirements of the Assessment Initiatives and by Whom 

Not applicable. 

Funding Available for the Last Three Fiscal Years, by Program Area if Possible 

Not applicable. 

A Brief Overview of Findings by Class Level 

Not applicable. 

Special Assessment Needs Identified in the State 

None. 

Contact: Richard Crofts 

Deputy Commissioner for Academic Affairs 
Montana Systems of Higher Education 
2500 Broadway 
Helena, MT 59620 
(406)444-6570 

rcrofts@oche.oche.montana.edu 




77 



81 



NEBRASKA 



Origins/Interest/Authority of Postsecondary Student Assessment in the State 

There is currently no statewide mandate for student learning assessment in this state. Institutions 
that are accredited by North Central have developed outcomes assessment plans and are in the 
process of implementing those plans. The Coordinating Commission requests information about 
student learning outcomes in its review of existing instructional programs. It also asks for out- 
comes assessment plans in proposals submitted by public institutions for establishment of new 
academic programs. 

Purpose of the Assessment Program 

There is no statewide student learning assessment for Nebraska. However, one of the purposes of 
seeking assessment data in the review of new and existing programs is to obtain evidence about 
program quality. The data are used to enlighten decisions about the viability of new programs 
that are proposed and about the continuation or discontinuation of existing programs. 

Description of Assessment Program 

There is no state-level assessment program. 

Validity and Reliability 

Not relevant. 

Common Data 

Not relevant. 

Reporting or Approval Requirements 

The only statewide assessment initiatives in Nebraska are under the responsibility of the Coordi- 
nating Commission for Postsecondary Education. The Coordinating Commission has expressed 
support for the outcomes assessment requirements of North Central. 

Funding 

Not Applicable. 

Overview of Findings 
Not Applicable. 

Special Assessment Needs 

No special assessment needs have been identified. 

Contact: Dr. Odus V. Elliott, Academic Officer 

Coordinating Commission for Postsecondary Education 

P. 0. Box 95005 

Lincoln, Nebraska 68509-5005 

(402) 471-0023 (phone) 

(402) 47 1-2886 (fax) 
oelliott@ccpe. state. ne. us 
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NEVADA 



Origins of Assessment Interest in Nevada 

The Board of Regents of the University and Community College System of Nevada adopted an 
assessment policy in 1989 that requires each campus to develop a plan of regular student educa- 
tional assessment. Initial plans were submitted to the Board in 1990. 

Description of the Initiative 

According to the policy, each campus is to assume responsibility for developing the assessment 
processes and procedures to be used. Plans are to be based upon campus mission and involve 
multiple assessment approaches. The policy also provides that plans are to reflect the mix of pro- 
grams and types of students at each school. While assessment approaches vary among institu- 
tions, the universities and community colleges are to work together to develop common 
approaches, where appropriate 

The Primary Purpose of the Assessment Program 

The primary purpose of assessment in the state is to stimulate curricular action and improvement 
at the institutional level. 

Brief Description of the Assessment Program 

Each campus has adopted a schedule of regular activities including a variety of surveys and inter- 
views which are conducted of students, graduates, faculty, employers, community members, and 
others. Results of college placement, graduate admission tests, and state and national licensure 
examinations are also being used to provide feedback on instructional programs, college services 
and employer satisfaction. Various class levels are utilized, depending upon the objectives. 

Progress is underway at the universities to assess the core curriculum. At one campus, in the 
mathematics area, a longitudinal study of students in the core curriculum math classes is under- 
way. Writing and critical thinking assessment continues, and systematic examinations of student 
learning in other areas of the general education curriculum are planned. A collaborative approach 
between a community college and university tracks and assesses transfer students through a writ- 
ing portfolio assessment program. 

The results of the academic program review process, which is undertaken by each campus and re- 
ported to the Board of Regents annually, are also being utilized to improve and enhance pro- 
grams. 

The Means Used to Establish Validity of the Assessment Instrument and the Reliability of 
the Process 

Each campus has developed a variety of survey instruments which attempt to link classroom and 
learning and the assessment methods. Longitudinal studies, systematic examinations of student 
learning in the core curriculum area, and assessment in the major areas include test scores, inter- 
views and focus groups. Two community colleges have initiated measurable student outcomes 
though Competency-Based Instruction programs. 
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Common Data or Test Results Collected Across the State 

The Board of Regents policy provides that the Chancellor’s Office, with the campuses, will de- 
velop appropriate measures of student persistence and performance, and will collect and monitor 
these data on a statewide basis. However, due to a lack of statewide resources, this is not being 
done at the present time. 

Reporting or Approval Requirements of the Assessment Initiatives 

Since the initial reports on campus assessment efforts were submitted to the Board of Regents in 
1990, the presidents of each campuses report, on a biennial basis, the results of their continuing 
assessment activities. 

Both the community colleges and the universities also address assessment requirements of their 
accrediting association (Northwest) during regular self-study and visitation activities. 

Funding Available for the Past Three Years 

Campuses fund assessment efforts with existing resources and current appropriations. No special 
funding particularly for assessment has been granted at the state level, and information is not 
available on individual campus expenditures by program area. 

Brief of Overview of Findings 

The creation of extensive and rigorous new core curriculum requirements for all students at one 
institution, and plans to review the general education curriculum at a second, will call for a com- 
prehensive examination of the impact of these changes in the future. Plans for the establishment 
of systematic examinations to evaluate students’ abilities in English, mathematics and critical 
thinking are underway. In addition, portfolio projects in specific majors related to writing and 
critical thinking skills are in progress at one university. 

Special Assessment Needs 

The multiple assessment approaches are based upon individual campus mission and goals. Due 
to limited resources and the need for campuses to focus on their unique mix of programs, special 
statewide assessment needs have not been identified. However, we are in the process of identify- 
ing potential systemwide institutional performance indicators for implementation over the next 
two-year period. 
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NEW JERSEY 



New Jersey does not have a state-level collegiate assessment program at the present time. More 
specifically, the previously-operational state-level Basic Skills Assessment Program (BSAP) and 
the College Outcomes Evaluation Program (COEP) that did employ common assessment instru- 
ments no longer exist. Moreover, state-level coordination of the higher education system has 
been restructured, with the Board and Department of Higher Education (which administered 
BSAP and COEP) having been replaced by the Commission on Higher Education and the Presi- 
dents’ Council. 

However, assessment and outcomes continue to be essential features of the New Jersey higher 
education system. There is now in place a set of guidelines for basic skills testing and remedia- 
tion, and another for institutional accountability reporting. While the latter does not currently call 
for learning outcomes to reporting, the guidelines in both areas are continuing to evolve along 
with the new coordinating structure. 

Contact: Philip L. Beardsley 

Director of Research 

New Jersey Commission on Higher Education 
CN-542 

Trenton, NJ 08625-0542 
(609) 292-4319 
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NEW MEXICO 



Authority for and Origins of Interest in Assessment 

The New Mexico Commission on Higher Education is a gubernatorially-appointed board author- 
ized by state statute to plan and coordinate postsecondary education. Among other duties, the 
Commission prepares recommendations for state funding of public postsecondary institutions. 
Although the Commission has historically considered assessment the prerogative of individual 
institutions, two recent statutory amendments have increased the Commission’s authority in this 
arena. A 1990 law requires the Commission to submit an annual report card to the governor and 
legislature, including "the results of [each] institution’s learner outcomes assessment program 
and steps taken to improve learner outcomes." A 1995 law authorizes the Commission to con- 
sider "educational outcomes assessments" when recommending funding for faculty salary adjust- 
ments. 

New Mexico has no statewide program for the assessment of college student learning. When the 
first state-mandated report card was prepared in 1990, the Commission found that institutional as- 
sessment programs varied widely; most institutions had little or no institution-wide assessment 
activities. For these reasons, and because a number of other, process measures dictated by the re- 
port card law have proven difficult to obtain in a reliable manner, the report card requirement has 
been frustrated. The Commission has therefore launched an effort to strengthen the capacity of 
institutions to measure student learning and to gather and use other measures of institutional per- 
formance. 

Current State Strategy and Action 

The Commission’s current philosophy is that — at least for the next few years — the state’s two 
dozen postsecondary campuses should each develop assessment programs tailored to their own 
missions and circumstances rather than being required to participate in any central, common as- 
sessment system. This belief was reinforced by an analysis of institutional plans for expanding 
outcomes assessment, submitted to the Commission in 1993. The analysis confirmed the dispa- 
rate state of activity and capacity to perform assessments across the campuses. Because the 
state’s regional accrediting body, the North Central Association of Colleges and Schools, was at 
the same time expanding its accreditation requirements to include assessment of student aca- 
demic achievement, the Commission chose to lend its support to that effort rather than develop a 
redundant or competing mandate. 

In 1 994, with financial support from the state legislature, the Commission awarded incentive 
funding to support two projects aimed at expanding assessment activities on campuses. One, con- 
duct of a statewide assessment conference for faculty and staff, was initiated in Spring 1 995 and 
will be continued in Spring 1996. Two spin-offs from this project are a statewide organization of 
campus assessment directors and initiation of a statewide newsletter on assessment. The state’s fi- 
nancial doldrums may limit the use of such incentives in the near future. 

The Commission has recently reactivated an improved statewide, unit record data base. The in- 
itial products of the data base are statewide student tracking studies, reporting rates of student 
persistence, transfer, and completion of degree programs. This information will be incorporated 
in the Commission’s annual report, The Condition of Higher Education in New Mexico. 
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In 1996, the Commission intends to organize advisory groups of institutional researchers and aca- 
demic officers to identify additional measures of institutional performance that might be col- 
lected and published on a statewide basis. 

Summary 

Although its legislature has recently strengthened state-level authority to conduct outcomes as- 
sessments, New Mexico has chosen to encourage institutional assessment activities that reflect 
the diversity of its postsecondary institutions. Outcomes measures such as graduation rates are 
being compiled and reported at the state level, but more direct measurement of student learning 
is currently left to the discretion of individual institutions. 

Contact: Dr. Bill Simpson 

Deputy Director for Educational Programs 

New Mexico Commission on Higher Education 

1068 Cerrillos Road 

Santa Fe, NM 87501 

Telephone: 505/827-7383 

fax: 505/827-7392 

e-mail: bsimpson@unm.edu 
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NEW YORK 



Origins of Assessment Interest in the State 

For decades, institution-based assessment has been mandated by regulation of the New York 
State Education Department. 

Description of the Initiative 

The regulatory mandate reads as follows: 

To be registered, each curriculum shall . . show evidence of careful planning. Institutional 
goals and the objectives of each curriculum and of all courses shall be clearly defined in writ- 
ing, and a reviewing system shall be devised to estimate the success of students and faculty in 
achieving such goals and objectives. The content and duration of curricula shall be designed to 
implement their purposes. 

For each curriculum the institution shall designate a body of faculty who, with the academic 
officers of the institution, shall be responsible for setting curricular objectives, for determining 
the means by which achievement of objectives is measured, for evaluating the achievement of 
curricular objectives and for providing academic advice to students. The faculty shall be suffi- 
cient in number to assure breadth and depth of instruction and the proper discharge of all other 
faculty responsibilities. The ratio of faculty to students in each course shall be sufficient to as- 
sure effective instruction. 

The regulatory mandate by the State Education Department affects all public, postsecondary in- 
stitutions in the two major systems — the State University of New York and the City University 
of New Yorkand all independent colleges and universities, as well as all the proprietary, degree- 
granting institutions in the state. In addition, CUNY — which includes 10 four-year colleges and 
seven community colleges — has required a basic skills test of all entering students since Septem- 
ber 1978. The test is for placement purposes only, and does not limit access. However, students 
who fail the basic skills test upon entry must pass it prior to their junior year. 

In 1994 the Board of Regents urged all institutions to develop their own indicators of perform- 
ance, to measure that performance, and to publish the results. The Board also directed Depart- 
ment staff to work with the higher education sectors in the State to develop indicators of the 
health for the entire system of 249 public, independent, and proprietary degree-granting institu- 
tions. That work continues in preparation for the next Statewide Plan for Higher Education. 

Primary Purpose 

The primary purpose of institutional assessment in New York is the statewide improvement of 
undergraduate education. 

Are Common Data or Test Results Collected Across the State? 

The regulatory mandate of the State Education Department is designed to foster assessment ef- 
forts within the institution. It is not designed to collect data for purposes of comparisons among 
the postsecondary institutions. 

Each Statewide Plan and interim Progress Report provides trend data by sector on enrollment, de- 
grees granted, remedial and developmental study and other indicators of higher education per- 
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formance. Graduation rate data is collected for all degree-granting institutions using a cohort sur- 
vival methodology. Completion rate information is collected for all non-degree proprietary 
schools and placement information by program is collected for all VATEA eligible institutions 
and for all non-degree proprietary colleges. 

Is Reporting or Approval Required of Assessment Initiatives? 

Results of assessment based upon the regulatory mandate are used primarily in program review 
and approval processes conducted by the Department of Education. 

Funding 

Institutions do not receive any specific or targeted funding for assessment. These activities are 
funded through the general institutional appropriation. 

Comments 

Assessment is expected to increase in importance in New York. The periodic reviews of institu- 
tions by the Education Department are beginning to focus more on "outcomes" as opposed to "in- 
puts." 

Contacts: 

Barbara W. Flynn 
Assistant Commissioner for 
Quality Assurance 

New York State Education Department 
Cultural Education Center - Room 5 A3 7 
Albany, New York 12230 
(518) 474-3896 



Thomas J. McCord 

Coordinator, Office of Research 

and Information Systems 

New York State Education Department 

Cultural Education Center - Room 5B44 

Albany, New York 12230 

(518) 474-6644 




85 



89 



NORTH DAKOTA 



The State Board of Higher Education seven-year plan directs each of the 1 1 campuses to have in 
place, by 1996, a comprehensive program to assess student achievement of learning goals as de- 
fined in the campus mission statement. This is interpreted to minimally be the assessment proc- 
ess required by the regional accrediting association. 

Additionally, the State Board has implemented a high school core curriculum admission require- 
ment for baccalaureate and graduate campuses and has contracted with ACT to study the impact 
of this requirement. Feedback is provided to the high schools. 

The University System is collaborating with other state agencies to develop a computerized track- 
ing system for following college/university students into the workplace. The system will enhance 
the assessment of college/university graduates’ educational experience for entry to and success 
in the workplace. 

Any further University System assessment activities will most probably incorporate the projects 
described herein. 

Contact: Gene A. Kemper 

Vice Chancellor for Academic Affairs 

North Dakota University System 

State Capitol - 600 E Boulevard Ave 

Bismarck ND 58505-0230 

(701) 328-2965 

(701) 328-2961 (Fax) 

e-mail: KEMPER@VMl.NODAK.EDU 
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OKLAHOMA 



The Constitution of Oklahoma charges the Oklahoma State Regents for Higher Education with 
responsibility for prescribing standards for admission, retention, and graduation applicable to 
each institution in The Oklahoma State System of Higher Education. The Oklahoma State Re- 
gents also have the responsibility to provide leadership in the coordination of the orderly transfer 
of students between and among institutions of the State System. Inherent in such responsibilities 
is the prescribing of mechanisms to monitor and facilitate the assessment of students for pur- 
poses of instructional improvement and State System accountability. 

Assessment in the state of Oklahoma has a multi-purpose objective. First and foremost, assess- 
ment is used to maximize student success by improving teaching and learning skills. Addition- 
ally, it serves an accountability purpose. Finally, student assessment assures the integrity of 
college degrees and other educational activities/goals, to increase the retention and graduate rates 
of college students, to enhance the quality of campus life in general, and to encourage high 
school students to improve their academic preparation for college. 

The Oklahoma State System for Higher Education uses a three part assessment system: entry- 
level, mid-level, and exit-level. Additionally, collegiate-level assessment can be linked to pri- 
mary/secondary level assessment for the purpose of improving student preparation. 

The following is a excerpt from the Oklahoma State Regents’ policy statement on student assess- 
ment: 

Each college and university shall assess individual student performance in achieving its pro- 
grammatic objective. Specifically, each institution will develop criteria, subject to State Re- 
gents’ approval, for the evaluation of students at college entry to determine academic 
preparation and course placement; mid-level assessment to determine basic skill competen- 
cies; exit assessment to evaluate the outcomes in the student’s major; and student perception 
of program quality including satisfaction with support services, academic curriculum, and the 
faculty. Such evaluation criteria must be tied to stated program outcomes and learner compe- 
tencies. 

An assessment fee has been approved which is designed to cover the cost of the assessment proc- 
ess. The fee is approximately one dollar per credit hour. Collection of assessment fees began in 
Fall 1991 and generates approximately 3 million dollars per year, roughly 20 dollars per student 
per year. 

Entry-Level Assessment 

The purpose of entry-level assessment is to assist institutional faculty and counselors in making 
decisions that will give students the best possible chance of success in attaining their academic 
goals. In Oklahoma, the ACT is used as the primary instrument for entry level assessment, and 
the State Regents have set minimum performance standards required for admittance to the State 
System of Higher Education. 

The minimum ACT score required for entry-level admission varies by the type of institution. At 
the two research universities in Oklahoma, students must score in the top 33 percent of ACT test- 
takers in the state. The ACT score equivalent to these percentages is based on the average of the 
preceding three years’ ACT scores for graduating high school seniors. Currently, this equates to 
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an ACT score of 21 or higher. At regional universities in Oklahoma, students must score in the 
top 50 percent of all Oklahoma seniors tested. This currently equates to an ACT of 19. Two-year 
institutions in the state have open admission. No minimum score is required, but the ACT or an 
equivalent standardized test must be taken for the purpose of placement. 

Each institution uses an established ACT score in the four subject areas of science reasoning, 
mathematics, reading, and English as the "first cut" in determining individual student readiness 
for college-level course work. The ACT subject scores provide a standard yardstick for measur- 
ing student readiness across the State System. The subject score requirements are communicated 
to college bound students, parents, and primary/secondary schools to make them aware of the 
level of proficiency recommended in the basic academic skills. Students scoring below these lev- 
els are required to remediate in the discipline area or, consistent with institution’s approved as- 
sessment plan, undergo additional testing to determine their level of readiness for college-level 
work. 

Institutional entry-level assessment programs include an evaluation of past academic perform- 
ance, educational readiness (mental, physical and emotional), educational goals, study skills, val- 
ues, self-concept and motivation. Student assessment results are used in the placement and 
advisement process to ensure that students enroll in courses appropriate for their skill levels. 
Tracking systems are used to link assessment information and course work completion records. 
These systems strengthen academic programs through evaluation and enhance student achieve- 
ment. 

Mid-Level Assessment 

Mid-level assessments normally occurs after the student has completed forty-five semester hours 
and prior to the completion of seventy semester hours for students in baccalaureate programs. 

For associate degree programs, assessment may occur at mid-level or at the end of the degree 
program. Examples of appropriate measures include academic standing, GPA, standardized and 
institutionally developed instruments, portfolios, etc. 

Generally, mid-level assessment tests competencies in the areas of reading, writing, mathemat- 
ics, and critical thinking. Thus, the results of mid-level assessment improve the institution’s gen- 
eral education curriculum. 

Program Outcomes Assessment 

Program Outcomes Assessment, or major field of study assessment, is the third component of 
the State Regents’ policy. Such assessments are designed to measure how well students meet in- 
stitutionally stated academic program goals and objectives. 

As with other levels of assessment, selection of the assessment instruments and of other parame- 
ters (such as target groups, when testing occurs, etc.) is the responsibility of the institution and is 
subject to State Regents’ approval. Preference is given to nationally standardized instruments. 
The following criteria are guidelines for the section of outcomes assessment methodologies: 
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a) Instrument(s) should reflect the curriculum for the major and measure skills and 
abilities identified in the program goals and objectives; 

b) Instrument(s) should assess higher level thinking skills in applying learned infor- 
mation; and 

c) Instrument(s) should be demonstrated to be reliable and valid. 

Nationally normed instruments required for graduate or professional study, or those that serve as 
prerequisites to practice in the profession, may be included as appropriate assessment devices. 
Examples are the GRE (Graduate Record Exam), NTE (National Teacher Exam), and various li- 
censing examinations. 

Assessment of Student Satisfaction 

The perceptions of student and alumni are important in the evaluation and enhancement of aca- 
demic programs and campus services. These perceptions provide an indication of the students’ 
subjective view of events and services which collectively constitute their undergraduate experi- 
ence. Evaluations of student satisfaction are accomplished via surveys, interviews, etc. Resulting 
data are to be used to provide feedback for the improvement of programs and services. 

Examples of programs and activities to be included in this level of assessment are satisfaction 
with student services, quality of food services, access to financial aid, resident hall facilities, day 
care, parking, etc. 

Primary/Secondary School Assessment/Accountability Feedback Systems 

In addition to assessment systems at the college level, there are 102 primary/secondary school 
districts (approximately 80,000 students) in the state which have voluntarily enrolled in the Okla- 
homa Educational Planning and Assessment System (OKEPAS). The purpose of OKEPAS is to 
help primary/secondary educators to better prepare their students for advanced levels of educa- 
tion. OKEPAS is a service of the ACT corporation, and the fee is currently paid by the Okla- 
homa State Regents for Higher Education. The program consists of two series of tests that all 
students at the participating districts are administered. 

The first OKEPAS test, EXPLORE, is administered in the 8th grade. EXPLORE is used to as- 
sess a student’s readiness for secondary education and to start students thinking about educa- 
tional decisions they will need to make in the years ahead. The second test is PLAN which is 
administered in the 1 0th grade. The results of this test can be linked with both the EXPLORE 
and ACT tests. Linkage to the EXPLORE allows student progress to be charted between the 8th 
and 10th grade. The PLAN test also acts as a baseline from which student progress can be 
charted between the 10th and 12th grade using the results of the ACT. Furthermore, the results 
of all of these tests can be linked with students’ college performance data using the Oklahoma 
State Regents’ Unitized Data System (UDS). UDS collects course level information on every stu- 
dent in the State System of Higher Education. This information is then relayed back to pri- 
mary/secondary schools to keep them informed of their students’ progress once in college. 

The Oklahoma Collegiate Success Profile provides feedback information to each high school site 
in Oklahoma. These profiles are made possible through a cooperative effort between the Okla- 
homa State Regents for Higher Education and the ACT corporation. The report tells each high 
school how many of their ACT-tested graduates enrolled in Oklahoma public colleges and the 
type of institution they attended. The report tells how many of these students persisted beyond 
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their first semester in college. This information is displayed for students who went to college 
with the required 1 1 units of core curriculum and those who matriculated without the core cur- 
riculum. The report also shows the rate at which students persist related to four ACT composite 
score ranges as well as information on high school grade point average and college grade point 
average. 

The Oklahoma State Regents for Higher Education produce a series of reports for every high 
school district in the state. Collectively, the three reports are titled The High School Indicators 
Reports. The first report provides information to high schools about the number and percentage 
of seniors who have matriculated at an Oklahoma state college or university. This number is 
used to calculate a high school to college going rate. The second in the series supplies each dis- 
trict and site with an ACT score based on the average performance of their students. Scores for 
the county and state are also provided for comparative purposes. The third report provides the 
first-semester average college GPA. Student performance is displayed in four GPA ranges, and 
by county, district and site. This system helps schools accurately assess how prepared their stu- 
dents are for college course work. 

Additional Elements of the Oklahoma Assessment/Accountability System 

College Report Cards. 

The Oklahoma State Regents are exploring the possibility of producing an institutional report 
card. The college report cards would be compiled for every institution in the State System for 
Higher Education. It will likely include such things as attrition/retention rates, graduation rates 
and job placement rates. These indicators could be calculated at both the institutional level and at 
the program level. 

The. Athletes Gmdmlktn Rais. Report 

The Athletes Graduation Rate Report is the result of a 1988 legislative mandate. It compares the 
graduation rates of all first-time entering students to those who have athletic scholarships. This 
report follows the methodologies set forth in the federal Student-Right-to-Know legislation. 

Each fall, student cohorts are built which consist of all first-time, full-time, degree seeking stu- 
dents. These cohorts are then tracked annually to record student progress. Graduation rates can 
be displayed by tier, institution, program, and race. . 

EloeemenL Rates. 

The Oklahoma State Regents have been working with the Oklahoma State Employment Security 
Commission to compute placement rates for graduates of the State System. This is accomplished 
through the use of blind social security number transfer systems. The placement rates are calcu- 
lated by program and include average wage information. Data sharing agreements are being 
worked out with several of the surrounding states so that students will be logged even when they 
receive employment outside the state. This project has been underway for one year now and will 
help colleges and universities assess the relative success of their students once placed in the job 
market. The data is still in a preliminary stage, and a report has not yet been produced. 
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Grade. In flation Study 

The Grade Inflation Study was conducted in 1994. The purpose of the study was to compare 
grade inflation in teacher education programs to other programs at the state and national level. 
Grades were analyzed in relation to historical trends and students’ ACT scores. Considerations 
are being made to continue this report in the future so that institutions can monitor their grading 
practices in relation to other institutions in the state. 

Conclusion 

Oklahoma has made great strides toward improving its assessment and accountability feedback 
systems over the course of the last ten years. This has been accomplished through the creation of 
a three stage assessment system, the linking of college level assessment with assessment at the 
primary and secondary level, and through the monitoring of such things as retention rates, gradu- 
ation rates, job placement rates and grade inflation at the collegiate level. 

Information is collected at all levels of education and is made available to those involved in the 
educational process for the purpose of improving the educational system in the state of Okla- 
homa. However, due to the lack of comparable data at the national level, it is still difficult to 
compare our state’s performance with that of the nation’s as a whole. It would be greatly benefi- 
cial to have NCES coordinate with the states to create common data elements relating to assess- 
ment and accountability measures. This information could then be collected by NCES and used 
for state level comparisons. 

Contact: Chancellor Hans Brisch 

Oklahoma State Regents for Higher Education 
500 Education Building, State Capitol Complex 
Oklahoma City, Oklahoma 73105 
Phone: (405) 524-9100 
Fax: (405) 524-9235 
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PENNSYLVANIA 



Origin/Interest/Authority of Postsecondary Student Assessment in the State 

Pennsylvania has no mandate for assessment. Secretary of Education Eugene Hickok is promot- 
ing raising the level of achievement at all levels of education, and is exploring ways to provide 
incentives for institutions of higher education to raise the standard for student performance. 

Primary Purpose of the Assessment Program 

Not applicable 

Brief Description of the Assessment Program 

Not applicable 

Means Used to Establish Validity 
Not applicable ... 

Common Data or Test Results 

Not applicable 

Reporting or Approval Requirements 

Not applicable 

Funding Available 

Not applicable 

Brief Overview of Findings 

Not applicable 

Special Assessment Needs 

Not applicable 

Comments 

Staff and members of the State Board of Education, the Department of Education, the institu- 
tions and the legislature are attempting to become better informed on assessment. 
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SOUTH CAROLINA 



Origins/Authority of Assessment in the State 

Assessment was initiated through the Commission on Higher Education in a broad policy docu- 
ment called The Cutting Edge, which was largely adopted into law in 1988 (See State Code of 
Laws Sections 59-104-650 and 59-104-660). Additional assessment requirements were added by 
the legislature in 1992 (State Code of Laws Section 59-101-350). 

Primary Purpose of the Assessment Program 

The primary purpose of the original (1988) program was and continues to be the improvement of 
educational quality, although accountability is clearly another purpose. Each institution is re- 
quired to provide the Commission with an annual report on the results of its institutional effec- 
tiveness program and the Commission must prepare a report for the legislature. The additional 
(1992) legislation requires a "report card" format and focuses more directly on accountability as 
its primary purpose, although questions arising from the data often cause the institutions to exam- 
ine issues of quality and improvement. The two required reports are combined into one docu- 
ment that is submitted to the legislature by January 15 of each year. Although legislative 
committees often raise questions about findings that are reported, there is no direct relationship 
between the results of the program and funding at this time. A proposal has been drafted that 
would tie a small percentage of higher education funding directly to performance indicators. 

Description of the Assessment Program 

Updated Guidelines for Institutional Effectiveness provide directions for implementation and re- 
porting under both the 1988 and the 1992 legislation. 

Under the initial (1988) program, which was phased in over four years and is institutionally 
based, the public colleges and universities use multiple methodologies to address assessment in 
areas required by the Commission as follows: (a) general education; (b) majors or concentra- 
tions; (c) performance of professional program graduates on licensing and certification exams; 

(d) program changes resulting from external evaluations; (e) academic advising; (f) entry-level 
placement and developmental education; (g) success of matriculants in meeting admissions pre- 
requisites; (h) achievement of students transferring from two to four-year institutions; (i) reten- 
tion and attrition; (j) minority student and faculty access and equity; (k) academic performance 
of student athletes; (1) student development; (m) library resources and services; (n) administra- 
tive and financial processes and performance; (o) facilities; (p) public service; and (q) research. 

Institutions maintain a plan on campus and at the Commission offices that can be modified and 
that defines each component (when necessary), indicates how it will be assessed, and provides a 
reporting schedule (not all components are reported on annually). The assessment of institutional 
effectiveness is a shared responsibility between administrators and faculty that must include 
tracking of student progress through the curriculum and follow-up of graduates. A specific goal 
is to "provide data . . . that can be used to initiate curriculum, programmatic, or policy changes 
within the institution." Therefore, emphasis in the annual reports on this part of the program is 
not only on findings, but on what actions the institutions will take or have taken to bring about 
improvement as a result of the findings. 
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South Carolina’s 1992 higher education report card legislation adds a requirement that the Com- 
mission report information and/or data in eleven specific areas (seven areas for two-year col- 
leges) so as "to easily compare peer institutions in South Carolina and other SREB states with 
the state’s public post-secondary institutions." Some of the eleven areas were already covered in 
the initial program, but the new legislation added the requirement for comparative formats and 
was generally more specific about data requirements. 

The performance indicators that must be reported on in this portion of the annual report include 
(a) number and percentage of accredited programs and programs eligible for accreditation;* (b) 
number and percentage of undergraduate and graduate students who complete their degree pro- 
gram;* (c) percent of lower division instructional courses taught by full-time faculty, part-time 
faculty, and graduate assistants;* (d) percent and number of students enrolled in remedial 
courses and the number of students exiting remedial courses and successfully completing entry- 
level curriculum courses; (e) percent of graduate and upper-division undergraduate students par- 
ticipating in sponsored research programs; (f) placement data on graduates;* (g) percent change 
in enrollment rate of students from minority groups and the change in total number of minority 
students enrolled over the past five years;* (h) percent of graduate students who receive under- 
graduate degrees at the institution, within the state, within the United States, and from other na- 
tions; (i) number of full-time students who have transferred from a two-year, postsecondary 
institution and the number of full-time students who have transferred to a two-year, postsecon- 
dary institutions;** (j) student scores on professional examinations, with detailed information on 
state and national means, passing scores over time, and the number of students taking each 
exam; and (k) appropriate information relating to each institution’s role and mission. 

There is an additional requirement that a survey on alumni satisfaction with the overall program, 
the major, and general education as well as of societal participation be conducted biennially of 
public institution graduates from three years prior. 

Validity and Reliability 

Inasmuch as measures of classroom teaching/learning are established by the institutions, and 
most use multiple methodologies to assess this area, the establishment of validity and reliability 
is specific to each method in general education or a major. The South Carolina Higher Education 
Assessment Network, a consortium of 48 institutions and two state agencies, is available to pro- 
vide technical assistance if needed. 

Common Data or Test Results 

Common data required under the 1992 legislation are described above. All institutions with stu- 
dents receiving athletic grants-in-aid also submit copies of their NCAA reports and limited sup- 
plemental information in response to item (k) of the original requirements. Common data are 
also collected on the percentage of students who meet admissions prerequisites and on minority 
enrollments, graduations, and faculty. Benchmarks are not pre-established, but results are re- 
viewed in all areas and specific recommendations for improvements are made as appropriate for 
the institutions’ missions. 

* Same for two-year colleges 
** Only two-year colleges report this item 



Reporting Requirements 

All 33 public institutions submit annual reports on institutional effectiveness to the Commission 
by July 1 of each year. The Commission compiles data and institutional summaries and develops 
a report that includes both tables and narrative information (300+ pages) for submission to the 
legislature by January 15. 

Notably, the narrative reporting process places responsibility on the institutions to describe how 
they interpret and use assessment data rather than just to report such data. This process is consid- 
ered a key component towards translating simple reporting of information into actions for im- 
provement. 

Funding 

Other than state support for the South Carolina Higher Education Assessment (SCHEA) Net- 
work, there is no special funding specifically designated for the institutional effectiveness pro- 
gram. The Network, which was initially funded through FIPSE, plays a major role in the success 
of the South Carolina program, sponsoring an annual conference and workshops, publishing a 
newsletter, maintaining a library of assessment publications and resources, and providing techni- 
cal assistance. 

Overview of Findings 

Findings on communication, problem solving, and thinking skills are generally distributed 
through reports on the assessment of general education in the 33 individual institutional reports 
in the Commission’s 1993, 1994, and 1995 Reports on Act 255 of 1992 and Summary Report on 
Institutional Effectiveness. It is clear that most of South Carolina’s public colleges and universi- 
ties consider communication and critical thinking skills to be among the goals of their general 
education programs. However, they use a variety of instruments and methods to assess these ar- 
eas, and it is extremely difficult to generalize with institutions that range from research universi- 
ties to two-year technical colleges. 

Special Assessment Needs 

While we agree that the postsecondary assessment needs identified in National Education Goal 
6, Objective 5 are critical, we believe that a much more broadly defined program such as that de- 
veloped in South Carolina and a number of other states provides both the institutions and the 
public with a better total picture of institutional strengths and weaknesses. We also believe that 
emphasis should be placed on the collection of data that provide diagnostic information and can 
be used for improvement rather than data that simply furnish a "rank" or "standing." 

Contact: Alan S. Krech 

Associate Commissioner 
SC Commission on Higher Education 
1333 Main Street, Suite 200 
Columbia, South Carolina 29201 
Phone: (803) 737-2291 
Fax: (803) 737-2297 

E-Mail: AKRECH@CHE400.STATE.SC.US 
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SOUTH DAKOTA 



1 . The South Dakota Board of Regents became interested in statewide assessment of academic 
programs in the early 1980s following the development of a program at Northeast Missouri 
State University. Program assessment began as a Board initiative with Board involvement 
in the selection of instruments and the design of the program. Campus control of assessment 
programs evolved over a number of years, with formal recognition of campus control occur- 
ring in 1987. In the 1990s, the Board’s requirement for assessment was merged with the 
North Central institutional requirements for accreditation. 

As part of an accountability initiative, the South Dakota Board of Regents is now interested 
in guaranteeing that each of the graduates of its six universities has attained at least a mini- 
mum level of proficiency in a number of areas. A pilot program will be implemented in the 
Spring of 1996 with a mandatory statewide rising junior examination to be implemented in 
the spring of 1997. Students who are not able to demonstrate proficiencies will not be able 
to continue to work toward a baccalaureate degree. 

2. The focus of program assessment is to provide information to the institution so that adjust- 
ments can be made in curriculum and the delivery of the curriculum to improve programs. 

The primary purpose of the student proficiencies program is to demonstrate that all bacca- 
laureate graduates have attained at least a minimal level of proficiency in a number of areas. 

3. The mechanisms for program assessment vary from campus to campus and from discipline 
to discipline. Programs are assessed in a variety of ways including nationally-normed objec- 
tive tests, faculty designed tests, capstone courses, portfolios, and student performances. 
Often, incoming students are initially assessed and then re-assessed at the end of the sopho- 
more year, but more commonly, in the senior year. While the methods of assessment vary, 
the emphasis is on assessing student outcomes and answering the following questions: Are 
students learning what the faculty think they are teaching? Have the students acquired the 
knowledge, skills, and abilities identified in the programmatic assessment plans? 

Student proficiencies will be assessed during the sophomore year. Current plans are that ob- 
jective tests and an essay will be administered to all sophomores. The objective tests will 
measure a variety of areas, including quantitative skills, problem-solving abilities, and scien- 
tific reasoning. 

4. Since faculty are directly responsible for the implementation of program assessments, they 
can directly link classroom teaching and desired learning outcomes with the program assess- 
ment method they select. 

The goal of student proficiencies is to assure that all students demonstrate a minimum level 
of proficiencies; therefore, student proficiency measures will be more directly tied to crite- 
ria references. 

5. Program assessments are campus-controlled and vary widely across the state. 

A single instrument will be used statewide to assess student proficiencies. Currently, the 
ETS Academic Profile and the ACT Collegiate Assessment of Academic Program (CAAP) 
instruments are being reviewed for adoption. Criterion referenced cut-off scores are ex- 
pected to be adopted based on the recommendations of a statewide committee. 

O 

ERLC 



96100 



6. The Board of Regents is expected to adopt student proficiency measurement requirements 
in December. The result of student proficiency examinations will probably become public 
documents that are used to report to the executive and legislative branches of government. 

7. Each campus has been given permission to implement a student fee to fund program assess- 
ment. 

A budget has not yet been approved for the student proficiencies initiative, but it is ex- 
pected to include a combination of resources from the Board, from student fees, and from 
institutional realignment of priorities. 

8. The campus program assessment findings are diverse and difficult to summarize. A copy of 
the reports provided by each campus to the Board of Regents in the Spring of 1995 is avail- 
able. 

The results of the student proficiencies initiative are not available since the program will en- 
ter pilot implementation in the Spring of 1996. 

9. No special assessment needs over and above those identified by the national education 
goals have been determined. 

Contact: Dr. Michel Hillman 

Director of Academic Affairs 
South Dakota Board of Regents 
207 East Capitol Avenue 
Pierre, SD 57501-3159 
Phone: 605-773-3455 
Fax: 605-773-5320 
Internet: mikeh@bor.state.sd.us 
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TENNESSEE 



Assessment plays a vital role in Tennessee Higher Education. The roots of the current assess- 
ments go back to the mid 1970’s when the Tennessee Higher Education Commission began to 
study the feasibility of basing some institutional funding on performance. That study led to the 
Performance Funding Program, first implemented in 1979 which has put more than a quarter of a 
billion dollars into institutional funding based on performance since that time. Total awards for 
the past several years have ranged from $25 million to $30 million each year. The standards un- 
der which funding is granted in this program are studied and altered as appropriate on a five-year 
cycle. The initial standards and all succeeding modifications were accomplished with full coop- 
eration and participation by campus and governing board personnel. Some parts of the standards 
have been in place from the beginning. These include standardized tests of performance in gen- 
eral education, accreditation of major programs, student and alumni surveys, and other such 
measures of educational quality. The current standards (1992-93 through 1996-97) and tables 
outlining the history of the program in terms of points awarded and dollars granted since the be- 
ginning are available from the state. 

With the passage of the state’s General Assembly in 1984 of the Comprehensive Education Re- 
form Act, assessment entered a new era. This legislation, in addition to addressing all areas of 
educational reform, specified certain "Legislative Benchmarks" on which the Commission would 
report each year. These benchmarks included standardized test scores, persistence to graduation, 
professional licensing examination results, library holdings, funding for research and public serv- 
ice, and job placement in vocational programs. These reports were mandated for a period of five 
years. 

In 1 989, the General Assembly adopted a document entitled Tennessee Challenge 2000 which es- 
tablished educational goals for the last decade of the century. Many of the reporting elements of 
the Legislative Benchmarks were included in the goals which were based loosely on the report of 
the Southern Regional Education Board’s publication, Challenge 2000. Reports of progress to- 
ward the goals established in this legislation are produced each year and distributed to the legisla- 
ture and the educational community. Several legislative acts passed since 1989 have expanded 
the content of this annual report. 

The primary purpose of all assessment in Tennessee is the improvement of the educational proc- 
ess. Most of the assessment effort is focused on undergraduate instruction. A side benefit of the 
assessment program is the increased legislative support which has resulted from such comprehen- 
sive reporting. 

The assessment system in Tennessee is best envisioned as wheels-within-wheels, or perhaps 
more aptly as gears-within-gears. Each campus has its own assessment program which, for the 
most part, focuses on educational improvement at the classroom level. These results feed into de- 
partmental and divisional reports which in turn become college and university reports. All these 
assessment results form a major part of each institution’s institutional effectiveness and planning 
cycles. Appropriate parts of institutional assessments flow into governing board planning proc- 
esses and a portion of those processes feed into the reporting system of the Tennessee Higher 
Education Commission. 
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Contact: 



Dr. Donald Goss 

Director of Academic Affairs 

Tennessee Higher Education Commission 

404 James Robertson Parkway, Suite 1900 

Nashville, TN 37243-0830 

Phone: 615-741-7565 

Fax: 615-741-6230 
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TEXAS 



Authority for Student Assessment 

In 1985, the Texas Higher Education Coordinating Board (THECB) appointed the Committee on 
Testing to determine how many Texas students entering public colleges were inadequately pre- 
pared for college-level work and to explore the feasibility of creating a basic skills test for these 
entering students. The recommendations of the Committee on Testing were sent to the Texas 
Legislature and during the 1987 session became law under Section 51 .306 of the Texas Educa- 
tion Code (TEC). THECB Rules and Regulations, Chapter 5, Subchapter P “Testing and Reme- 
diation” implement the Texas Academic Skills Program (TASP). 

Primary Purpose of Assessment Program 

The primary purpose of the TASP is the diagnostic assessment of the basic skill preparedness of 
entering students for college-level coursework in the areas of writing, reading, and mathematics 
with the intent to improve undergraduate education and to improve retention to graduation. Test 
results are sent to a student’s college or university. In the case of a failing score on a tested sub- 
ject, the student is placed in mandatory remediation. Diagnostic scores for the tested skills and 
subskills are included with the score report. 

Description of Assessment Program 

TEC 51.306 requires that all students entering a public institution of higher education in the fall 
of 1989 and thereafter take a reading, writing, and mathematics basic skills examination prior to 
the accumulation of nine or more semester credit hours (SCHs) or the equivalent. Multiple- 
choice testing is used in all tested areas, though the writing test results are based mainly on a 
written essay. The examination cannot be used for admission purposes. If skill deficiencies are 
identified, the student is required to participate in continuous remediation until the student mas- 
ters all sections on the examination. The student may not progress to upper division work be- 
yond 60 SCHs nor graduate from any Texas public institution of higher education without first 
passing all portions of the TASP Test. Institutions are required to offer advising programs for all 
students and remedial programs for students with demonstrated skill deficiencies. 

Validity and Reliability 

Content committees are composed of college personnel selected from across the state to review 
the test sections for content validity. Tested skills reflect preparedness to attempt college-level 
coursework. Passing criteria are set by the THECB. 

Common Data and Test Results 

All public institutions of higher education report on their remedial programs through the CBM- 
002 TASP Report to the THECB. Results are reported in the Annual Effectiveness of Remedia- 
tion Report. Successfully completing remediation is determined by successfully passing all 
sections of the TASP Test. Students requiring remediation are compared against students not re- 
quiring remediation for grade earned in first college-level mathematics and English classes, 
grade point average, retention and highest award earned. 
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Reporting Requirements 

TEC 51 .306 requires all institutions to report to the THECB on the effectiveness of remedial and 
advising programs. The THECB publishes statewide reports. 

Funding 

There is no direct state funding for the testing program, which is paid through the student test 
fee. Two hundred thousand dollars is appropriated each biennium for student test fee waivers. 
Approximately $500,000 is appropriated to the Texas Higher Education Coordinating Board 
each biennium to administer the program. The assessment program results in additional enroll- 
ments in remedial courses and non-course-based instruction. 

Overview of Findings by Class Level 

Not appropriate. 

Special Assessment Needs Identified 

None. 

Contact: . Dr. David Gardner 

Texas Higher Education Coordinating Board 

PO Box 12788 

Austin, TX 7871 1 

Phone:512-483-6150 

FAX: 512-483-6147 

E-mail: gardnerdd@thecb.texas.gov 
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UTAH 



Origins 

A policy advocating institutional assessment to improve educational quality was approved by the 
State Board of Regents in its 1986 Utah System of Education Master Plan. In 1992 the Utah Leg- 
islature put into law a statute mandating biennial reports on student assessment, faculty produc- 
tivity, and program and facility measures. 

Purpose 

The primary purpose is to determine how well the System is meeting its goals with regard to edu- 
cating students. The assessment report was/is delivered to the Board of Regents and the legisla- 
tive Education Interim Committee. The report is used to assess the quality of the education 
enterprise and determine what other indicators would provide a fuller picture of accomplish- 
ments and impact of the System on the State. Individual institutions are expected to report to the 
Regents their implementation of assessment procedures. 

Description 

The assessment is done biennially. A 1992 legislative mandate required assessment of students 
which includes these indicators: admission scores, progress, and outcome measures. Also re- 
quested are resource management indicators. These include faculty productivity, as assessed by 
the number of hours spent teaching and in teaching-related activities, scholarly pursuits, research 
funding, and on- and off-campus service. Measures of program effectiveness and facilities man- 
agement are requested. These include program review and accreditation reports, library staffing 
and acquisition data, and utilization of space information. The System added a section on the 
Higher Education Technology Initiative. The progress of transfer students will be added for the 
1997 biennial assessment and accountability report. 

These data are gathered from State Data books and questionnaires completed by the individual 
institutions. 

Establishment of Validity 

In the two previous statewide assessments, student progress was measured by retention and 
graduation rates, employment information, and acceptance to graduate school. It is unlikely that 
student scores will be included. Thus, there is no attempt to assess the teaching/learning transac- 
tion. However, as the System’s general education curriculum moves to a competency based sys- 
tem, methods of assessment will be developed and employed to determine if competencies are 
learned and applied. The findings from the assessment are validated through triangulation; other 
data sources support the findings. 

Test Results 

Not applicable at this time. 

Requirements 

The assessment results are reported to the legislature and regents biennially. The Office of the 
Commissioner oversees changes in development of the instrument, collection of data, and inter- 
pretation of analysis. 
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Funding 

No additional funding has been appropriated. Staff time is used to refine the instrument, collect 
and analyze the data, interpret the findings and write the report. 

Overview 

Learning outcomes on communication, problem solving, and thinking skills were not assessed. 
Once competencies within the general education curriculum are adopted statewide, assessment 
instruments will be developed so that the competencies can be measured. 

The assessment process is updated following each biennial report. A statewide taskforce reviews 
the previous document, identifies areas that need to be included and/or altered, and revisits the 
methodology and instrument used to gather assessment data. The taskforce includes repre- 
sentatives from academic affairs, student services, institutional research and the business commu- 
nity. All institutions in the System are represented. 

Needs Identified 

The Utah Assessment and Accountability report does not include actual student test scores in spe- 
cific courses. However, it does include other student data such as the number of terms taken to 
completion and the number of credits carried per term. It also includes student pass rates on na- 
tional examinations, graduation rates, employment rates and rates of acceptance into graduate 
programs. Besides these indicators, the current assessment report includes faculty productivity 
measures, program accreditation and review data, and facilities utilization data. The impact of 
the System’s Higher Education Technology Initiative is also included. The next biennial report 
will include the initiative’s effect upon enhanced learning and student/faculty development. 

Contact: Phyllis "Teddi" Safman 

Assistant Commissioner for Academic Affairs 
State Board of Regents 

355 West North Temple - 3 Triad Center #550 
Salt Lake City, Utah 84180-1205 
(801)321-7101 
(801)321-7100 FAX 
psafman@cc.utahsbr.edu 
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VERMONT 



Origins of Assessment in Vermont 

Vermont has no statewide governing board and there have been no legislative mandates regard- 
ing assessment of student learning. Assessment efforts are largely directed by the higher educa- 
tion institutions and their own boards of trustees. Within Vermont, the Vermont State Colleges 
and the University of Vermont function as distinct public corporations, each with their own 
board of trustees. 

Purpose of the Assessment Program 

The primary purpose of the initial efforts to assess student learning within the Vermont State Col- 
leges is to improve undergraduate education. Assessment information is used internally to iden- 
tify points of concern and to identify curricular and student support areas which require further 
inquiry. 

The Assessment Program 

The five colleges within the VSC currently assess entering students’ skills in three areas: writing, 
reading, and mathematics. The results of these tests are used in student placement within the vari- 
ous curricular areas, not for admissions testing. In addition, the VSC tracks student retention and 
program completion. 

Assessment Instruments 

The VSC currently does not use an instrument that links classroom teaching with assessment of 
student learning on a systemwide basis. 

Common Data Results 

Vermont has no system to report common data or test results. 

Reporting Requirements 

There are no statewide reporting requirements. However, within the Vermont State Colleges, the 
institutions undergo programmatic self-study and review by the VSC Board of Trustees on a 
regular five-year cycle. Special program reviews may also occur outside this cycle. 

Funding 

There are no appropriations related specifically to assessment activities in Vermont. 

Findings 

None. 

Special Assessment Needs 
None at this time. 
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Contact: 



Dr. Joseph B. Moore 

Director of Planning and Academic Affairs 
Vermont State Colleges 
P.O. Box 359 

Waterbury, Vermont 05676-0359 
phone: 802-241-2520 
fax:802-241-3369 
e-mail: moorej@quark.vsc.edu 
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VIRGINIA 



Origins of Assessment Interest in the State 

The legislature initiated an assessment policy in 1986 when it directed all public institutions in 
the state "to establish assessment programs to measure student achievement" in Senate Joint 
Resolution 83. 

Primary Purpose 

The primary purpose of the state’s assessment initiative is to stimulate curricular action and atten- 
tion at the institutional level. In 1986, we hoped that the same mechanism could also serve to 
make institutions accountable to the public. 

Description of the Initiative 

The legislative mandate directed the State Council of Higher Education in Virginia to spearhead 
the effort by establishing assessment guidelines in cooperation with institutions in the state. 

As a result, SCHEV published assessment guidelines in April 1987. The guidelines encouraged 
institutions to use multiple indicators of assessment, such as: 

• Direct measures of student learning, including standardized tests and new faculty-developed 
assessment measures (portfolios, senior projects, examinations, etc.). 

• Existing information, such as that pertaining to admissions, retention, and graduation rates; 
community college transfer rates; licensing and certification examinations; job placement; and 
alumni satisfaction. 

• Basic skills testing and evaluating the success of remediation 

Institutions needed at least to assess students in the major and general education, survey alumni, 
assess the success of remediation, and provide information on their graduates’ success to feeder 
high schools and community colleges. The guidelines allowed the institutions to develop their as- 
sessment plans, but a gubernatorial directive established in May 1987 that institutions would be 
eligible for incentive funding for the 1988-90 biennium only if they had an "adequate student as- 
sessment plan." By June 30 of that year, less than two months later, all institutions had submitted 
an assessment plan to the SCHEV. Although some required some revision, all were eventually 
judged adequate. 

Based on the institutional budgets, funds averaging $12 per full-time student were granted to the 
institutions to implement assessment procedures. The institutions submitted progress reports in 
1988, followed in 1989 by full reports on the results of the first biennium of assessment. The 
state now enters in 1995-96 its fourth full biennium of institutional assessment. Reports for this 
biennium will be made orally. 

Summary results of the institutional assessment efforts have been incorporated into the biennial 
Virginia Plan for Higher Education. The plan reports that a wide scope of assessment activity 
has been undertaken, with overall enthusiasm, generally strong commitment to the process, and 
results that have been used, in many cases, to improve curricula. 
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The Means Used to Establish Validity 

Each individual campus is responsible for establishing the reliability and validity of its measures. 

Are Common Data or Test Results Collected Across the State? 

There is no common collection of data, although efforts to track retention and success rates of 
students transferring from two-year to four-year schools have resulted from assessment. 

Is Reporting or Approval Required of Assessment Initiatives? 

Originally, all assessment plans were approved by the SCHEV, working with a group of external 
consultants, in the latter half of 1988. Annual progress reports do not require formal Council ap- 
proval, but are reviewed and commented on by SCHEV staff. Biennial reporting of results are re- 
viewed by Council staff and consultants from the institutions. 

Funding 

In 1988-90, the state appropriated $4.4 million ($2,962,100 from the general fund) for the bien- 
nium for institutional assessment. This money was, in the 1990-92 budget, incorporated into the 
institutions’ base budgets, and institutions in many cases made significant additional contribu- 
tions. But since the institutions have seen their general-fund appropriations decrease by over 
$400 million since 1989, that money arguably has disappeared. 

A Brief Overview of Findings 

Given the program-by-program nature of the findings, they cannot be summarized. One general 
finding of the programs is that general-education programs lacked coherence or point and, as a 
consequence, could not be assessed. Virtually all have been revised to focus on learning goals as 
a result. 

Special Assessment Needs Identified in the State 

While the legislative agenda initially turned on public accountability, the assessment agenda as 
pursued by SCHEV is primarily focused on the improvement of undergraduate teaching and 
learning. This has left the accountability needs unmet. The Council has attempted to fill the gap 
through its Indicators of Institutional Mission project, which provide indirect measures of institu- 
tional input and output. It may also be necessary to have some general direct measure of student 
learning. In cooperation with a number of other SREB states, the Council is looking into a test of 
general intellectual skills that may be given to a sample of students across the state. 

Contact: Margaret A. Miller 

Associate Director for Academic Affairs 
State Council of Higher Education 
James Monroe Building 
101 North Fourteenth Street 
Richmond, Virginia 23219 
(804) 225-2627 
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WASHINGTON 



Origins of Postsecondary Student Assessment 

In 1987, the Washington State Higher Education Coordinating Board (HECB) Master Plan chal- 
lenged the public two- and four-year institutions of higher education to develop a multi-dimen- 
sional program of performance evaluation. The plan envisioned assessment as a link between 
two separate but complementary goals: to improve the quality of undergraduate education and to 
provide needed information about student outcomes to the HECB. Four initial means of assess- 
ing educational quality were identified: 

• Follow-up data on graduates’ work experience 

• Satisfaction surveys about students’ educational experiences 

• Surveys of employer satisfaction with employees’ college preparation 

• Pilot tests of nationally normed sophomore year tests of communication, computation, and 
critical thinking skills 

After three nationally-normed tests (the College Measures Program, Collegiate Assessment of 
Academic Proficiency, and Academic Profile) were piloted, the HECB concluded on the basis of 
a task force report and evaluations by faculty members that the tests were not the best tools to as- 
sess the quality of undergraduate education. The Board modified the assessment plan to include 
six common components: 

• Collection of entry-level baseline data 

• Intermediate assessment of quantitative and writing skills 

• End-of-program assessments 

• Program review 

• Alumni satisfaction surveys 

• Employer satisfaction surveys 

The HECB initiative was reinforced by the Governor and the Legislature, which provided 
$400,000 for assessment activities to each of the six four-year public institutions, and to the State 
Board for Community College Education in the 1989-91 biennial budget. This funding was sup- 
plemented in 1990 by $60,000 for each of the 27 community colleges. (Five technical colleges 
have subsequently been added to the system.) 

Primary Purpose 

The overriding goal of assessment has been to help faculty and institutions improve their pro- 
grams, processes, and policies in light of student learning in degree programs, writing, quantita- 
tive skills, and critical thinking as well as other student outcomes (e.g., employment). The Board 
encouraged each institution to develop its own approach to assessment, including adopting differ- 
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ent organizational structures, ways of implementing assessment, tests, and models for assessment 
activities. Assessment has flourished due to its emphasis on faculty-driven assessment questions 
and methods and its primary use as a means for self-improvement. 

Assessment has been useful to institutions in allocating state resources, setting and measuring the 
accomplishment of program or institutional objectives, and prioritizing among competing budget 
uses. Both of these functions — assessment as improvement and aid to budget allocations — are ex- 
pected to continue to be crucial in the future. 

Brief Description of the Assessment Program 

The six components listed previously serve as the common framework for the assessment pro- 
gram. Assessment efforts are designed by and tailored to the needs of each institution. 

Means used to establish validity and reliability 

Not applicable. 

Common Data or Test Results 

Not applicable. 

Reporting or Approval Requirements of the Assessment Initiatives 

An Assessment Coordinator appointed by each institution coordinates the assessment activities, 
and acts as liaison to the HECB. Until 1995, institutions reported biannually to the HECB; now 
only annual reports are required. 

The Board stresses the use of assessment results in the preparation of institutional budget re- 
quests. (The HECB reviews, evaluates, and makes recommendations to the Legislature on operat- 
ing and capital budget requests from four-year institutions and the community and technical 
colleges.) 

Funding Available 

State appropriations for assessment during the 1993-95 biennium totaled $2,232,000 shared 
equally among the six four-year public institutions ($372,000 each) and $3,725,000 shared 
among the State Board for Community and Technical Colleges and the 32 community and tech- 
nical colleges. The HECB receives no direct funding for assessment. 

Brief Overview of Findings by Class Level 

Examples of activities that have occurred on campuses, particularly in the areas of communica- 
tion, problem-solving and thinking skills are available in the annual assessment report. 

Special Assessment Needs identified in the state over and above those identified in the Na- 
tional Education Goals 

The HECB is in the process of developing competency-based admissions standards to eventually 
replace the current HECB minimum admissions standards. The new standards will be designed 
to be consistent with the K-12 educational reform work of the Commission on Student Learning, 
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to signal more precisely to students what it means to be prepared for college, and to allow stu- 
dents who have followed various pathways in high school to assess and demonstrate their readi- 
ness for the baccalaureate learning experience. A process for assessing the success of students 
admitted under these new standards will need to be designed. 

Contact: Kathe Taylor, Ph.D. 

Senior Policy Associate 
Phone: 360-753-7815 
Fax: 360-753-7808 
E-mail: kathet@hecb.wa.com 
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WEST VIRGINIA 



West Virginia colleges and universities have been formally involved in institutional and system- 
wide assessment activities since 1987 when the West Virginia Board of Regents named a Task 
Force on Assessment. This group, comprised of representative members from the College and 
University President’s Council, the state-wide Advisory Council of Faculty, the state Academic 
Dean’s Council and Central Office personnel, developed a set of “Principles Which Govern and 
Undergird Institutional and System Assessment Efforts” and “Institutional Guidelines” for cam- 
pus assessment activities. During the ensuing academic year all higher education institutions pre- 
pared an inventory of entry-level, interim, and post-assessment instruments and activities while 
the Assessment Task Force identified common instruments used by system schools and summa- 
rized the nature and extent of assessment data being collected on the campuses. The Task Force 
gathered and distributed definitional information regarding assessment and detailed descriptions 
of successful assessment programs and sponsored four institutional pilot assessment projects 
most of which focused on entry level assessment and subsequent placement activities. 

In 1989 the West Virginia Legislature realigned the governing structure for higher education in 
the state. The Board of Regents was dissolved and separate governing boards were established 
for the University System and the State College System of West Virginia. The Task Force on As- 
sessment was re-named the West Virginia Higher Education Council on Assessment and contin- 
ued to include membership from both systems. The Chairperson was retained but the 
composition of the Council was changed to include a representative of each public college and 
university. This Council’s initial charge was to develop a set of recommendations for each gov- 
erning board establishing institutional responsibility for the development of assessment plans 
and to develop a mechanism for the exchange of campus assessment data. Such recommenda- 
tions were developed and adopted by both boards in the fall of 1989. The first recommendation 
adopted by both boards stated that “Each public college and university is urged to develop a five- 
year comprehensive assessment program which is compatible with its mission and educational 
objectives. Preliminary focus should be on learning outcomes.” Soon campus assessment com- 
mittees were established and the state-wide Council on Assessment initiated a series of work- 
shops designed to facilitate campus efforts. Initial workshops were informational sessions 
regarding CAAP, COMP, WORKKEYS, and BASE conducted by ACT and Riverside Publish- 
ing personnel. “Critical Components of the Campus Assessment Program,” (1991) a workshop 
focusing on the development of assessment models appropriate for 2-year, 4-year, and graduate 
institutions featured critiques by Dr. Trudy Banta, then Director of the Assessment program at 
the University of Tennessee, Knoxville. Additional programs have included a Classroom Assess- 
ment Workshop (1993) featuring Dr. Thomas Angelo, Director of the Academic Development 
Center at Boston College, and a fall 1994 workshop led by Dr. Rita Meyer, Vice-Chancellor for 
Academic Affairs at the University of Minnesota, Crookston, regarding North Central Associa- 
tion guidelines for effective assessment programs. 

The Council on Assessment, in cooperation with the state college Academic Deans Council, de- 
veloped and approved a set of guidelines for the exchange of assessment data between campuses. 
Thus, two-year colleges may receive information regarding the academic performance of their 
graduates enrolled at public baccalaureate institutions and 4-year schools may also obtain infor- 
mation regarding the performance of their graduates in the state’s public graduate programs. 
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A primary goal of the state-wide Council is to facilitate campus assessment planning. Thus, early 
on the Council prepared and distributed a brochure, “Assessment: The Faculty Role.” The group 
routinely provides information regarding current assessment research and conferences. The 
Council Chairperson and representative members have participated in regional and national con- 
ference programs sponsored by AAHE, ETS, SAIR, and the North Central Association in order 
to share information regarding proposed standards, strategies and concerns pertinent to assess- 
ment. 

The Assessment Council has recently developed a Higher Education Graduate Survey Form de- 
signed to gather collective data from all students graduating from public colleges and universi- 
ties in West Virginia regarding their perceptions of the quality of their undergraduate preparation 
as well as information about their success in securing employment in-field following graduation. 
The form has been recommended for implementation in the spring of 1 996. 

Although the focus of both governing boards has been upon the development of autonomous 
campus assessment programs, each has provided unique directives regarding expected institu- 
tional progress and areas of focus. The University System Board adopted a series of Assessment 
Initiates in 1992 which emphasized its commitment to assessment and its general expectations 
for institutional involvement and progress. University System institutions annually prepare a de- 
tailed report of their assessment activities as they relate to current assessment focuses determined 
by the Chancellor and the Board. The University System has also provided workshops, training 
sessions, and follow-up activities regarding Total Quality Management. 

The State College System has focused its attention on the assessment of general education. The 
1990 Board of Director s Plan for Quality and Accountability requires that “all freshmen enter- 
ing state college system schools must demonstrate proficiency on standard tests in math, reading, 
and English composition in order to be placed in college-level English and math courses.” Sub- 
sequently, the Board of Directors of the State College System adopted ACT cut-offs for entering 
freshmen in these subjects. Such standards remain in effect. 

The West Virginia higher education governing boards and the state-wide Council on Assessment 
have acted primarily to encourage and facilitate assessment activities rather than to issue direc- 
tives regarding assessment planning. However, the West Virginia legislature has recently passed 
two bills which include mandates for extensive higher education assessment activity. The First, 
Senate Bill 412 (1992), the state report card legislation, directs the Higher Education Central Of- 
fice to collect, analyze and report data regarding higher education in order to “make information 
available to parents, students, faculty, staff, state policy-makers and the general public on the 
quality and performance of public higher education.” The report, developed by the Central Of- 
fice Research unit in concert with a highly representative state-wide report card committee, is a 
comprehensive document which reveals both individual campus and systems’ progress and suc- 
cess. Among the specific indicators reported are the number of students served by tuition and fee 
waivers, the average ACT and SAT scores of entering freshman, the number of degrees awarded, 
the ratio of students to faculty, fiscal support for higher education, the number of applications 
and acceptances to medical schools, and the number of medical school graduates in particular 
spending areas. Of particular note is information regarding institutional retention rates. 

Senate Bill 547, passed during the 1995 legislative session, outlines further requirements for in- 
stitutional and systems’ assessment. The bill mandates that “the governing boards shall prepare 
institutional report cards for institutions under their jurisdiction and system wide report cards ...” 
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and that “In assessing progress toward meeting goals and in developing trend information, the 
governing boards shall review report card data in relation to previously adopted board goals, five- 
year plans, regional and national higher education trends and the resource allocation model.” In 
addition to calling for data on academic performance, the legislation stresses the need for assess- 
ment of citizenship and human relations skills. The bill requires the development of institutional 
and system strategic plans which focus on program review and fiscal accountability. Such plans 
were recently provided to Central Office and governing board personnel for analysis and review. 

West Virginia governing boards have allocated approximately $15,000 annually for state-wide 
assessment programs and materials. However, the primary responsibility for funding assessment 
activity has been borne by the campuses. In 1993 the State College System Board provided a 
one-time allocation of $5,000.00 for assessment activities for each campus under its purview. 
Systems’ Chancellors have facilitated campus fiscal planning through their endorsement of a 
group ACT-CAAP pilot project which provided reduced fees for test instruments to all partici- 
pants. 

West Virginia public colleges and universities have made substantial progress in the develop- 
ment of comprehensive assessment plans which are compatible with their missions and re- 
sources. Review of the campus programs suggest that they are characterized by strong 
administrative support and faculty involvement, the designation of appropriate assessment instru- 
ments and activities to assess learning and personal growth and commitment to utilization of as- 
sessment data to improve curricular offerings. Currently campus personnel are attempting to 
integrate assessment activity into institutional planning processes while the state-wide focus is 
on attempting to achieve a broader perspective regarding assessment by gathering and examining 
collective data indicative of the preparation of the total state student population. 

Contact: Dr. Suzanne T. Snyder, Chairperson 

West Virginia Higher Education Council on Assessment 
(304)-367-4219 
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WISCONSIN 



The University of Wisconsin System, with 15 institutions serving over 150,000 students, 
adopted new accountability measures in 1 993 on the recommendation of a special task force con- 
vened by Governor Tommy Thompson. The task force, composed of a diverse group from the 
private sector, state government, the University of Wisconsin (UW) Board of Regents, faculty, 
staff, and students, began its work by defining accountability as "being answerable for the effec- 
tive discharge of the mission of the organizations." 

Following a review of the system’s mission statements, the group identified three areas of par- 
ticular concern to stakeholders: (1) delivering a high-quality undergraduate education, (2) meet- 
ing the needs of business and other organizations in Wisconsin and (3) being customer-oriented 
and responsive to the concerns of customers, particularly students and parents. 

Focusing on these priorities, the UW Board of Regents adopted a core set of performance indica- 
tors to demonstrate accountability in each area. In developing the indicators, regents were guided 
by the task force’s recommendation that measures should be based on outcomes and should in- 
clude a limited number both qualitative and quantitative indicators. 

The 1 8 key indicators, which build on a more limited set of indicators already in place at the Uni- 
versity of Wisconsin, are used to measure the performance of the system in seven general areas. 
These include effectiveness, efficiency, quality, access, diversity, stewardship of assets, and con- 
tribution to compelling needs of the states. 

The following sample from the UW system’s 1994 report on "Accountability for Achievement" 
describes key indicators, their corresponding performance goals, and progress as of September 
1994: 

• Faculty share of undergraduate instruction. The goal is to increase faculty involvement in 
undergraduate instruction. Progress: Almost 70 percent of undergraduate instruction is deliv- 
ered by faculty, down slightly from 70.4 percent in fall 1992; 23 percent is delivered by in- 
structional academic staff, up from 22.3 percent in 1992; 5.6 percent is delivered by graduate 
teaching and assistants, up slightly from 5.5 percent in 1992. 

• Sophomore competency test. The goal is to exceed the national average in writing and mathe- 
matical skills. Progress: "UW students score above national average in writing and math 
skills" as measured by the American College Testing-Collegiate Assessment of Academic Per- 
formance. This was the first year that UW students took the test; the test will be repeated in 
the 1997-98 academic year. 

• Minority student enrollment and graduation rates. The goal is to increase minority student 
enrollment and graduation rates. Progress: Enrollments for African-American, Asian-Ameri- 
can, Hispanic, and American Indian students increased by 4 percent between 1992 and 1993, 
while minority graduation rates showed an overall 3 percent increase (comparing the eventual 
graduation rates of the group that began in 1986 with the group that began in 1987). 
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• Facilities maintenance. The goal is to reduce the maintenance backlog on institutional facili- 
ties. Progress: "Building commission has approved funds to reach 32 percent of maintenance 
goal" as part of a ten-year plan to eliminate the $364 million maintenance backlog identified 
in a building condition survey. The UW board has requested $126 million for 1995-97 to keep 
the plan on schedule. 

• Employer satisfaction with University of Wisconsin graduates. The goal is to improve the 
career-readiness of graduates and the system’s responsiveness to Wisconsin businesses and 
professions. Progress: "Employers rank UW graduates high" according to a survey of Wiscon- 
sin businesses and industry. The survey asked employers to rate graduates on basic skills (91 
percent responded good or excellent) professional knowledge (83 percent responded good or 
excellent), and critical thinking skills (91 percent responded good or excellent). Future sur- 
veys will provide a means for charting progress over time. 

In addition to monitoring and publishing its performance measures, the UW system and each of 
its institutions have developed processes (or refined existing ones) for enforcing accountability. 
Among them are more explicit guidelines for post-tenure review, regular evaluations of all fac- 
ulty and staff members, student evaluations of courses, and making summaries of the course 
evaluations generally available to aid students in course selection. 

Contact: Daniel P. Layzell 

Director of Policy Analysis 
University of Wisconsin System 
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PUERTO RICO 



The Origins 

A major priority of the Council on Higher Education in Puerto Rico is the establishment of a sys- 
tematic, statewide, formal assessment program though, at this time policies for such a program 
have not been formulated. The economic feasibility of implementing such an assessment pro- 
gram is currently under investigation. The need for assessment programs, however, has been rec- 
ognized by the Council on Higher Education as well as by both private and public institutions of 
higher learning in Puerto Rico. 

Primary Purposes 

The primary purpose of assessment is the determination of the effectiveness of instructional pro- 
grams in achieving their expressed outcomes. Knowledge of such outcomes aid in suggesting 
curricular modifications, including the development of new educational programs which will 
challenge students and increase the relevance of these instructional programs to the needs of soci- 
ety. Established assessment programs, for example, seek to reduce student dropout rates and en- 
hance student achievement. 

Brief Description of Assessment Programs 

Assessment is conducted on different levels in institutions with assessment programs: the class- 
room level; the program level; the office or department level; and the institutional level. Survey 
assessment is conducted in many of the institutions with assessment programs by using one or 
more of the following approaches: 

1 . Portfolio Approach - Provides a continuous assessment of student achievement and faculty 
effectiveness by collecting samples of student work for a specified period of time (e.g., a 
semester, a year, etc.); 

2. Reflective Diary - Students make notes on what has been studied during the week, reflect 
on what has been studied, and discuss with the professor what was studied to determine if 
the material presented has been understood. 

3. Repetition - Students are asked to repeat in their own words the statements made by the 
professor to determine if they have understood what was taught. 

4. Three-Minute Paper - Students answer three questions about what was taught: what he un- 
derstood, what he was unable to understand, and the aspects of the class he thinks need im- 
provement. 

The survey assessment approaches described above are designed to assess basic, not complex, 
student skills. 

Measures Used to Establish Validity 

Angelo and Cross’ Classroom Assessment Techniques: A Handbook for Faculty is used to estab- 
lish validity and reliability in some institutions where assessment programs are in place and to 
provide continuous and systematic evaluation of the efficacy of the assessment techniques used. 

Student attainment of established objectives and the efficacy of instructional practices used to 
achieve these objectives is assessed via the systematic collection of pertinent information. Such 
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goals are attained through the systematic adherence to a well-defined evaluation blueprint, the 
collection of information on the practices essential to the achievement of these goals and a com- 
mitment to continuous and systematic examination of the outcomes of instruction. Data yielded 
by such commitments form the basis for assessing instructional outcomes with educational out- 
comes. 

Common Data of Test Results Collected Across the States 

Statewide procedures for the systematic collection of data and program assessment have not 
been established. Through the Puerto Rico State Postsecondary Review Entity regulation proc- 
ess, efforts were made to develop quantitative measures of effectiveness among postsecondary in 
stitutions. An Island-wide data collection system was designed and its implementation is still 
under consideration. 

Improvement of professor instructional skills is a common objective in institutions with assess- 
ment programs in place. Student evaluation of professors is a technique commonly used for this 
purpose. After the student evaluation of the professor has been discussed with him, it is filed for 
a limited time and eventually discarded. 

Reporting of Approval Requirements of Assessment 

Program review and approval of accreditation and licensure procedures applicable to public and 
private institutions are used to assess programs in Puerto Rico. At the institutional level, individ- 
ual professors may undertake assessment projects but such assessments are not mandatory. As- 
sessment initiatives requiring financial support, however, must receive prior approval from the 
program director. 

Funding Available for the Past Three Years 

Funding, though very limited, is available on a recurring basis for assessment programs in place 
from unit implementing the assessment. 

A Brief Review of Findings by Class 

Communication skills, problem solving abilities, and critical thinking abilities are not specifi- 
cally included in assessment programs in institutions with such programs. Student attitudes, satis- 
faction, and like variables are emphasized in assessment programs currently operating in Puerto 
Rico. 

Special Assessment Needs Identified 

Critical thinking, problem solving, and communication are among the higher skills for which the 
development of assessment instruments needed in assessment is under consideration in some of 
the institutions engaged in student evaluation. 

Comments 

The data reported here are based on a limited survey of the assessment practices employed in 
some private and public institutions of higher education in Puerto Rico. Generalization of these 
findings to a broader population is unwarranted. Nonetheless, the elements required for the initia- 
tion of a statewide assessment program appear to be in place in certain institutions. 
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All institutions of higher learning in Puerto Rico are accredited. Most of them are accredited by 
the Middle States Association of Colleges and Schools (MSA), and are required to have assess- 
ment programs in place prior to receiving such accreditation. The Council on Higher Education 
is contemplating the establishment of a master plan which will include the need for assessment 
as an instrument for improving higher education in Puerto Rico. 

Contact: Dr. Eusebio Diaz-Diaz 

Planning Division 
P.0. Box 19900 

San Juan, Puerto Rico 00910-1900 
Phone (809) 724-7100 Ext. 2061 
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WHAT POLITICAL LEADERS EXPECT FROM 
POSTSECONDARY ASSESSMENT 



Charles S. Lenth 

Education Commission of the States 

Interest by this nation’s political leaders in postsecondary student assessment has been apparent 
during the last ten years. Indeed, some observers have argued that much of the initial impetus as 
well as the continuing motivation behind the assessment movement came from the broader inter- 
est in student "outcomes" and public accountability, as manifest in the calls of governors and ac- 
tions of many state legislatures beginning in the mid-1980s. At the same time, it must be 
acknowledged that this political interest in assessment has been episodic, focused on quite differ- 
ent objectives, and often out-of-sync with the preferred practices and principles for assessment 
within colleges and universities. This provides a dynamic tension within assessment between the 
internal and external, between improvement and accountability, between formative and summa- 
tive uses, and between those doing it and those who need to know. 

My thesis in this brief paper is that just as political and other external expectations have shaped 
assessment in the past decade, so these same forces will affect the purposes and directions of col- 
legiate level assessment in the coming years — and appropriately so in my view. It is important, 
therefore, to examine these external expectations for assessment to see where they come from, 
how they differ, and what types of responses are most likely to be effective. It also seems useful 
to explore these external expectations for assessment as a basis for speculating about what forces 
are likely to shape collegiate-level assessment in the next few years. At least among state-level 
political leaders, several major policy issues and societal concerns are likely to push their interest 
and support for assessment in new and fairly clear directions in the coming years. 

External Expectations in the 1980s 

The wide dissemination of several reports by the Southern Regional Education Board, the Educa- 
tion Commission of the States, the National Governors Association, and other policy-oriented or- 
ganizations in the early and mid 1980s put governors and legislators at the forefront of those 
calling on colleges and universities to assess the learning outcomes of students. These, to be 
sure, were not the only voices and motivators, but they were among the most prominent. Some 
institutions such as Alverno College in Wisconsin were already demonstrating approaches to 
comprehensive, institution-wide assessment. Tennessee had established limited performance 
funding that built in several types of state-wide student outcomes assessment under a rubric of 
performance "criteria." Georgia and Florida, followed by several other states, established or set 
out develop some form of state-wide postsecondary student testing, either for admission/place- 
ment in college level courses or as a gateway, particularly for transfer students, to junior stand- 
ing in baccalaureate programs. 

For the most part, these precursors of what became a vigorous assessment movement were not 
aimed at student assessment per se, but developed or adopted forms of assessment for other pur- 
poses. Early institution-based assessment at Alvemo and a small number of other institutions 
was part of an alternative model for a more integrated, student-centered and skill-based approach 
to undergraduate education. In Tennessee, assessment was instituted not as a way to look at stu- 





dent outcomes as much as a technique for allocating the small amount of non-formula-based 
funding on the basis of some measure of institutional performance. In those states that estab- 
lished general testing programs, the purposes were grounded in concerns for student preparation 
and readiness for college-level work. For political leaders, these were issues of basic educational 
standards and fiscal accountability. In any case, these and other external motivators blended into 
the assessment movement when it began to expand in the mid- 1980 s. 

State Assessment Initiatives. Primarily in response to a perception that many public institu- 
tions, particularly large research universities, were disregarding teaching and learning in under- 
graduate education, several states undertook major initiatives to shift institutional resources and 
attention in that direction. It is important to note that these state-level initiatives were motivated 
by conditions both internal and external to higher education for which assessment was only one 
of several policy approaches. New Jersey and Illinois illustrate this in different ways. 

New Jersey initiated state-wide, college-level assessment in a more direct and comprehensive 
manner than any other state. With the support of then Governor Tom Kean and substantial in- 
creases in funding from the legislature, Chancellor Ted Hollander and the Board of Higher Edu- 
cation initiated a major effort to enhance the stature and quality of New Jersey public colleges 
and universities. In combination with multi-year competitive challenge grants and strengthening 
institutional autonomy and governance, the College Outcomes Measurement Program (COMP) 
provided for the development of multiple forms of student assessment and program evaluation, 
part institution based and part state based. One component required the regular evaluation of the 
effectiveness of all postsecondary remedial programs, with an eye toward improving these so 
that all enrolled students could move quickly into college-level course work. The early results 
from this evaluation and assessment identified not only the extend of remedial course work at 
public colleges and universities, but the substantial variation in costs and effectiveness. The state 
department also funded the development, by ETS and with substantial faculty involvement, of a 
capstone college graduation assessment, the General Intellectual Skills (GIS) assessment. Based 
on set of contextual performance assessments and administered to a sample of college seniors, 
GIS was intended to as a tool for institutions to assess and improve their undergraduate educa- 
tion outcomes and for the state to evaluate the comparative performance of public colleges and 
universities. Following initial development, the results from the first GIS assessment proved to 
be unpalatable to institutional leaders and others. When changing economic conditions undercut 
the financial support for this comprehensive institutional improvement initiative and state politi- 
cal leaders became less supportive, the entire set of state initiatives was terminated and set aside. 

In Illinois, the Board of Higher Education adopted policies requiring institution-based student as- 
sessment as part of an broad initiative to improve undergraduate education. This initiative was an 
attempt by the board and its staff leadership to "get ahead of the curve;" that is, to put into place 
policies and a process of change to help move public colleges and universities to address some 
important external, public concerns before the pressure from outside became more vociferous. 
The Illinois initiative was, however, considerably less ambitious and more decentralized than in 
New Jersey. In particular, the undergraduate assessment component was to be institutionally 
based and embedded in the established program review process. This approach tended to inhibit 
institution-wide assessment of undergraduate outcomes, and produced very little comparative 
performance data at the state level. 
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The approaches and results in both New Jersey and Illinois were much more complex — and am- 
biguous — than these brief overviews convey. My purpose is only to draw some tentative observa- 
tions about the dynamic between the external, political forces and the directions of collegiate 
assessment at the state level. First, whether quite forcefully as in New Jersey or more subtly as in 
Illinois and a number of other states, the purposes of state-level assessment initiatives were de- 
fined primarily by or in anticipation of the needs in the external environment of higher educa- 
tion. In retrospect, the differences across states were mainly those of strategy and stage of 
development, not those of longer-term objectives. State based assessment itself has been a forma- 
tive rather than a summative process, with objectives that emerge and are clarified as a result of 
the accumulation of experience and the evolution of new techniques. This developmental meta- 
phor it seems to me is important to thinking about the purposes of assessment in the future, and 
its relations to externally defined needs and expectations. 

My second, related observation is that one must also view the past and future of assessment in 
terms of its practical effectiveness, if you will its developmental strategy. Here too, as with think- 
ing about its objectives, the external context and political environment are very important — per- 
haps even more so than realized at the time. The comprehensive and ambitious assessment 
initiative in New Jersey was terminated when state financial support was no longer available, 
and when political support was undermined by institutional opposition, among other factors. The 
more incremental Illinois approach lived on, not because it was any more successful in estab- 
lishing assessment per se, but because it led to several other strategic steps. Arguably, at least, 
the more decentralized and incremental assessment policies adopted in Illinois provided the time 
necessary for the state agency to establish its roles in certain core assessment areas and for the 
state political leadership to become comfortable with the necessity and directions for change in 
higher education. Specifically, the Illinois Board of Higher Education had sufficient time to in- 
volve the institutions in the development of state-level student assessment data systems (both a 
high school feedback report and a postsecondary student unit record system), and to embed as- 
sessment in a much more comprehensive and consequential initiative in the early 1990s, the 
Board’s Priorities, Quality and Productivity initiative. 

Patterns and Discontinuities. Looking back at the dynamic between assessment and the exter- 
nal environment during the 1980s, it is more helpful to look for patterns than discontinuities. The 
episodic nature of political leaders’ interest in and support for assessment is clear; differences 
across states and periods are even more obvious. More interesting, however, is to look for pat- 
terns in the objectives and in the types of support political leaders seem willing to provide. With- 
out going into detail, it seems to me state political leaders have demonstrated a pattern of support 
not so much for assessment in particular, but for using assessment and other techniques as ways 
to get colleges and universities to examine more closely how well they are meeting student and 
societal expectations. This interest has been relatively constant for at least the last decade, al- 
though as I will argue below it evolves and becomes more focused to reflect what those external 
needs are. 

Political leaders’ strategic and developmental interest in assessment is even more difficult to 
grasp and grapple with. But looking back in this area as well I think one can discern some pat- 
terns along with the obvious discontinuities. Political leaders often do not understand the com- 
plexities and intricacies of developing new ways to assess the effectiveness and outcomes of 
college and other postsecondary education. While there are some obvious examples of misunder- 
standing and impatience, it seems to me there is a pattern of fairly consistent interest and support 
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for the exploration and development of new assessment methods. Conflict has resulted, it seems 
to me, not as much from the impatience of political leaders as from the lack of responsiveness 
from the higher education community to this developmental agenda. Those political leaders who 
have advocated and supported assessment have fairly consistently recognized that substantial 
technical and infrastructure development will be needed. Their response, by and large, has been, 
let us get on with these tasks. This is a position in favor of assessment, not in opposition, al- 
though it does throw the challenge of further development back on higher education leaders and 
the assessment movement. 

Looking to the Future — What Will Political Leaders Expect? 

To the extent that political leaders, particularly at the state level, shape the purposes and support 
the development of new forms of postsecondary student assessment, how will they attempt to in- 
fluence the directions of assessment in the future? My answer is impressionistic at best, even 
more impressionistic and speculative than interpretations of these influences in the past decade. 
To the best of my knowledge there are no wide scale survey data or other hard evidence on the 
attitudes and expectations of political leaders toward assessment. Nor it is a topic that is foremost 
in their minds or prominent in their speeches. My tentative observations are based on some un- 
systematic interviewing and focus group conversations with various state political leaders over 
the past year or two, more extensive and personal contact with a small number of governors and 
legislators, and some exposure to their public speeches, press accounts and other sources. These 
limitations notwithstanding, I will argue that state political leaders are now and will increasing 
become interested in how postsecondary assessment can and will link to three larger issues. 

Linkages and Responsiveness to K-12 Standards. The most compelling of these is how to link 
collegiate level assessment with the related and already more powerful changes affecting K-12 
education. These potential linkages are visible to political leaders in several ways. First, although 
the National Education Goals adopted by the collective action of the then-President and state 
governors in 1989 are no longer given much attention, the standards-based-education movement 
and other reform efforts both leading to and pushed forward by this unusual collective political 
action are very much alive. The notion of setting goals and monitoring progress in education has 
yielded to a more sophisticated agenda-setting and implementation approach involving: 

1 . a process for establishing standards that reflect leaning and performance expectations for 
students at various levels; 

2. the development of new, more authentic ways to assess and give credit for achieving these 
outcomes; and 

3. the first exploratory steps toward building these standards and new assessments into the edu- 
cation system through restructuring the curriculum, teacher education and professional de- 
velopment, school performance indicators and other means. 

The realization, even piecemeal, of this systemic reform effort in K-12 education will require 
substantial adjustment and specific types of support from higher education. These adaptations 
are already underway in Oregon, Florida, Wisconsin and other states in initiatives to change col- 
lege admission policies to be more parallel to standards-based K-12 education, to reform teacher 
education and professional development, and to redesign college curricula and teaching practices 
to be more congruous with current student expectations. Perhaps even more important, in the 
minds of political, business and community leaders who have been involved in these education 



reforms, the logic of this reform agenda appears to apply equally well to higher education. If ex- 
plicit student learning and performance standards work well in the secondary environment, why 
don’t these apply to postsecondary education equally well? When we have new means to assess 
performance against these standards, will assessment at the collegiate level take on more of these 
authentic, performance-based forms? When we have more explicit standards and advancement 
through demonstrated performance up through high school and to the point of college admission, 
can we apply this same logic to determining student progress through college degree programs 
as well? 

These are among the questions and insights that are beginning to come to the minds of political 
leaders and others. A very prominent example in the "P-16 Initiative" of Governor Zell Miller in 
Georgia, in which the state higher education system leadership is a primary partner. Less visible 
but more numerous are the regional' K- 16 initiatives at several sites around the country, and the 
substantial involvement of higher education faculty in the development of K-12 standards. 
Clearly, there are many implications and roles for postsecondary assessment in addressing the in- 
itial linkage issues and, eventually, the more fundamental restructuring challenges of standards- 
based reforms. Comprehensive standards frameworks are still a good way off for K-12 
education, and surely farther yet for postsecondary education. I believe, however, that the pres- 
sures and rationale for collegiate assessment to be explicitly linked to performance standards and 
assessments at other levels will continue to grow. Before higher education sees systemic stand- 
ards reform, it is very likely to see innovative institutional leaders take up the call of political 
leaders to design and provide models of standards based postsecondary education, with new 
types of assessment being key to this. 

Value-For-Money Assessment. A second expectation that political leaders appear to hold for as- 
sessment is that it should address questions of cost effectiveness — what student outcomes are 
achieved at how much cost. This represents a blending of the demand for fiscal accountability 
for the expenditure of public funds with the value-added framework for pre- and post-assessment 
of the learning that takes place during college. As the concept of value-added assessment be- 
comes better know, this connection is made by political leaders without realizing that techniques 
are not yet available to measure either the educational gains or the precise costs involved, let 
along to combine these into a value-added-per-dollar-expended assessment. Despite this, the ex- 
pectation seems to be growing that such an assessment framework would be desirable in the fu- 
ture. Both the sharply rising costs of college education and the constraints on public tax support 
and subsidies lead in this direction. 

Given these expectation and the likelihood of an increasingly constrained fiscal environment, 
state political leaders will continue to push assessment in this direction. One way to attempt to 
address this expectation is through developing some indirect linkages between financial data and 
educational outcomes. Private colleges and universities already do this in informal ways, arguing 
that while college "X" costs more, what students get for their money in four years is worth the 
cost. Public institutions tend to argue only the low cost side, without much information about 
what student get for their investment, how long it will take, and what long-run returns are likely. 
Despite the technical difficulties and lack of comparative statistics, I believe that political leaders 
and others are likely to push college assessment in this direction of cost -effectiveness. 



Assessing Learning Outside the Classroom. A third and growing influence on assessment will 
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involve assessing learning that occurs outside as well as inside the classroom and campus. To 
many governors and legislators, this seems quite obvious and necessary. Witness the agreement 
by the western governors in November 1995 to explore the potential for a multi-state "virtual uni- 
versity" that could award credit for learning no matter where it occurs. Other governors are com- 
mitted supporters of workplace and applied learning options, beginning with high school 
students but extending logically to postsecondary learning as well. In either case, the scope and 
functions of assessment need to change substantially to include forms and locations for learning 
outside of traditional classroom environments. 

A colleague put the challenge this way. Suppose that we are consultants to the founding presi- 
dent of a new university. He or she wants our advice on the needs of this new university, how it 
should be set up and, specifically, what kind of student assessment system to establish for admis- 
sion and student advancement. The only thing the founding president currently knows is the loca- 
tion of the new university; it will be located right alongside the information superhighway. In 
this location, much of the available knowledge base and learning resources will be out on the in- 
formation highway, rather than in the traditional classroom setting. Students will come with spe- 
cific education and training needs to augment the knowledge and learning gained elsewhere. 
Faculty will be oriented to the world-wide resources, not to those held on campus. Libraries will 
be connecting points, not stable collections. And assessment of student learning may be more im- 
portant and challenging that ever before, since unless it takes into account the knowledge and 
learning from elsewhere, the new university’s contributions to student learning will be small and 
marginal indeed. 

Individuals from outside higher education, and specifically many political and business leaders, 
seem to grasp immediately the implications of this new university. This may be the world in 
which they already function, and where they see the future. To accommodate this world, student 
assessment and the granting of credentials and degrees will need to take into account the large 
and growing proportion of learning that takes place outside the conventional boundaries of class- 
room and campuses. Assessment will no longer be classroom and campus bound, but will need 
to be linked across physical locations and with the learning and resource networks of the world. 
How fast and how far assessment will be able to move in these directions cannot be certain, but 
the reasons and pressures to. move are already being articulated by political leaders and will con- 
tinue to be felt both within and from outside the traditional academic framework. 

Concluding Thoughts 

In this challenging and rapidly changing context, assessment of postsecondary student learning 
needs to take on broader and more central roles for colleges and universities. If not, like those 
very institutions, what we now know as collegiate assessment will become increasingly separate 
and marginalized in relation to the spread of knowledge and learning in society. Looking back at 
the last decade of development, the interest and pressure to integrate assessment into the class- 
room and throughout the campus has been much stronger than the external determination to es- 
tablish new forms of external assessment. I believe that this is in the process of changing. The 
long-standing tension between the internal and the external, between formative and summative 
uses, and between those who do assessment and those who need to know the results is about to 
enter a new era. The healthy and creative tensions will not go away, but the external forces will 
become more compelling than ever. 

Given this environment, higher education needs to prepare for and participate in the develop of 
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more independent, comprehensive and authentic types of learning assessment that will augment, 
but not necessarily replace, the assessment that now occurs in classrooms, across campuses and 
programs, and to a lesser extend across states. Some political leaders are already reflecting these 
future needs as they push higher education to pay closer attention to questions of standards and 
measurement of student learning at all levels, when they asked now unanswerable questions 
about value-for-money, and when they plan virtual universities around or in place of traditional 
institutions. Assessment, long disregarded and frequently marginalized, may turn into one of the 
core functions of postsecondary education in the future. This is the direction in which many po- 
litical leaders and other external influences are and will continue to push postsecondary educa- 
tion. 
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PERSPECTIVE FROM THE NATIONAL EDUCATION GOALS PANEL 



Ken Nelson, Executive Director 
National Education Goals Panel 

Thank you for your attendance and willingness to tackle tough and critical issues. Special thanks 
to Sal Corrallo for his untiring efforts to enhance the postsecondary experiences of all students. 

Einstein said, "We can not begin to solve the complex problems we face today with the same 
level of thinking we had when we created them." A new level of thinking is required of this 
workshop, of you and your state to solve the challenges of postsecondary assessment. It was a 
new level of thinking that created the National Education Goals. 

Gov. Romer, a member of the Goals Panel but speaking as a parent, has often asked, "Where do 
I go to find the appropriate, quality postsecondary education for my children? Where can I find 
the right information about what value respective postsecondary institutions will add to my 
child’s learning?" How would you answer the Governor’s question? 

Role of the Goals Panel 

The role of the Goals panel is to monitor and report on progress the nation and states are making 
toward achieving the national education goals. The Panel would like your assistance to achieve 
Goal 6, Objective 5, and that s what this workshop is all about; to determine how we can best as- 
sess, monitor and report on postsecondary academic achievement. 

Goal 6 reads "By the year 2000, every adult American will be literate and will possess the knowl- 
edge and skills necessary to compete in a global economy and exercise the rights and responsi- 
bilities of citizenship." 

Objective 5 of Goal 6 reads: "The proportion of college graduates who demonstrate an advanced 
ability to think critically, communicate effectively and solve problems will increase substan- 
tially." 

History of the Panel’s Recommendations 

I. As you perhaps know in July 1990 the Goals Panel was created to develop a mechanism to 
monitor and report on the nation and states’ progress toward meeting the goals. Tradition- 
ally the Panel has selected "some of the best minds" in the country to form a Resource 
Group which makes its recommendations to the Panel. 

A Resource Group on Adult Literacy and Lifelong Learning was formed which included 
Mark Musick as the convener and Peter Ewell and Paul Barton as members. Its recommen- 
dations on Goal 6 indicators were presented in a discussion document to the public in re- 
gional forums. The Discussion Document was then finalized on March 25, 1991 and 
entitled Measuring Progress toward the National Education Goals: Potential Indicators and 
Measurement Strategies." 

These germinal recommendations were made: "Assessing the Knowledge and Skills of 
Graduating Seniors. The Resources Group notes that if the National Education Goals Panel 
wishes to assess directly the ability of college graduates to think critically, communicate ef- 
fectively and solve problems, a new kind of assessment will have to be created. (The Re- 
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source Group considers both the NALS and the Graduate Record Examinations to be inap- 
propriate for this purpose). Suggesting that developing such an assessment (which could be 
modeled on the NAEP) would be both complex and controversial, the Resource Group esti- 
mates investment costs of "several scores of million of dollars and 5 years or more of devel- 
' opment work for the system to become operational." 

"A NAEP-like examination of graduating college seniors would be necessary if the Panel is 
serious about measuring the advanced thinking, communication and problem-solving skills 
of the Nation’s college graduate." (In November, 1991 The National Center of Education 
Statistics convened a study design workshop to examine means of assessing college student 
learning in support of the sixth National Education Goal.) 

II. In February 1992 a new Task Force was formed to focus more precisely on "assessing the 
national goal relating to postsecondary education." This time Mark Musick and Charles 
Lenth were among the advisors and David Longanecker among the members. Part of its 
charge was to investigate and report on "the feasibility and desirability of a sample-based 
collegiate assessment which would provide regular national and state representative indica- 
tors of college graduates’ ability to think critically, communicate effectively, and solve 
problems." 

In a report published July 31, 1992 the Task Force made six recommendations to the Panel: 
1) encourage the development of a sample-based national system of standards and assess- 
ment for postsecondary education; 2) suggest that content and performance standards be de- 
veloped for general cognitive skills, higher order thinking skills, and occupational specific 
skills where appropriate; 3)insist that in order to maximize their usefulness, assessment ef- 
forts be better coordinated through a formal structure; 4) urge the Secretaries of Education 
and Labor to approve funding for assessment and skills certification activities only if the ac- 
tivity is coordinated and recorded in the inventory of assessment activities; 5) recommend 
the creation of a separate coordinating council for postsecondary standards and assessment 
that parallels those recommended by the National Council on Education Standards and Test- 
ing for elementary-secondary education and recommend financial support from the Con- 
gress to support this activity; 6) establish as an objective the development of a constellation 
of indicators of postsecondary performance which includes basic skill levels, occupational 
skill levels and higher order skills. 

In summary, the Task Force concluded that a postsecondary assessment system would be 
both desirable and feasible, and recommended that content and performance standards be 
developed for general cognitive skills, higher order thinking skills, and occupational spe- 
cific skills where appropriate. (In November, 1992 NCES convened a second study design 
workshop to discuss which skills should be assessed in a collegiate assessment, how to se- 
lect standards, and other measurement issues.) 

III. In April and May of 1993 the Goals Panel conducted a series of regional hearings in order 
to obtain input and public comment from higher education representatives, faculty, stu- 
dents, etc. In July 1993 the Goals Panel made its recommendation on Postsecondary Assess- 
ment: "The National Education Goals Panel believes that it is both feasible and desirable to 
develop a national sample-based postsecondary assessment system that will provide regular 
national and comparable state indicators of college graduates’ ability to think critically, 
communicate effectively and solve problems and that includes assessments of occupation- 
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specific skills for students in occupationally-specific programs." Therefore, these general ar- 
eas of concern were identified and incorporated in a draft resolution. 

1. A national sample-based system of assessment for postsecondary education should be 
developed. Such a system should encompass the differing institutional missions within 
the postsecondary sector. This national assessment system will not provide information 
about the performance of any individual postsecondary institution or student. 

2. To promote the highest levels of performance throughout postsecondary education, the 
assessment system should be driven by rigorous content and performance standards. 

3. In assessing students’ abilities to think critically, communicate effectively and solve 
problems, the system should be designed to reflect students’ differing fields of study 
and occupational areas. 

4. A broad-base, consensus building process should be used to set appropriate standards 
and achievement levels for this postsecondary assessment system and to review and 
evaluate assessment approaches. The consensus building process should involve fac- 
ulty and administrators representing a variety of institutions (varying in mission, geo- 
graphic location, etc.) as well as employers, policymakers, institutional researchers, 
assessment experts, and higher education coordinating boards. 

5. A variety of regularly reported postsecondary performance indicators should be devel- 
oped from this assessment system. They should be chosen for their ability to provide 
useful information to different audiences including policymakers, system participants 
(e.g., administrators, faculty and students) and the general public." 

IV. The Goals Panel was hopeful that NCES would take it the next step. As our invitation to 
this conference indicated in its background information, "In 1993 NCES prepared a Request 
for Proposal that focused upon skill identification for the college graduate. The purpose was 
to identify and obtain a consensus of the skills and levels of competency needed by college 
graduates beyond college. The results were to be used as a basis for the development of a 
national survey of college student learning. A primary use of the information was to be used 
on the status of efforts to meet the national education goals. The RFP was not funded due to 
budget constraints and it is unlikely that NCES will soon have sufficient resources to con- 
duct a comprehensive national study, as originally envisioned. As a result, alternative strate- 
gies were developed." And that brings us to this conference. 

V . I want to illustrate what indicators for Goal 6 we report and how we do our reporting. This 
might be helpful as we think about state and local reports. 

1995 DATA VOLUMES: The National Data Volume, page 118-132, Goal 6 indicators, 
with Exhibits 56 and 57 provide direct measures of the Adult Literacy; and Exhibits 58 and 
59 on Workforce Skills. Exhibit 60-62 provide direct measures of the Objective, Opportu- 
nity to Acquire Knowledge and Skills; and Exhibit 62 updates us on Postsecondary Enroll- 
ment and Completion. On pages 183-186 "The Technical Notes and Sources," provide 
background information. 

The State Data Volume provides for each state the direct measures of the Goal on Adult Lit- 
eracy and Citizenship as well as a direct measure of the Objective on Postsecondary Enroll- 
ment. The supportive Technical Notes and Sources are found on pages 256-258. 



ERjt 



129 



133 



1995 CORE REPORT, page 23, core indicators 16, 17, 18 provide information on Adult Lit- 
eracy, Participation in Adult Education and Higher Education; graphically presented on 
pages 47-49. In addition, you might find this interesting. On page 57 we present our Na- 
tional Data Collection Schedule and on page 60 our State Data Collection Schedule docu- 
menting a dramatic need for more data at the state and national level. 

1 995 EXECUTIVE SUMMARY, page 8 provides information on the progress we are mak- 
ing on three indicators for Goal Six: Adult Literacy, Participation in Adult Education, and 
Participation in Higher Education. 

IN CONCLUSION. During its entire history the Goals Panel has demonstrated its deep and 
abiding commitment to provide the best data possible on the nation and states’ progress to- 
ward realizing Goal achievement. But it is apparent that not enough data is available to ef- 
fectively report on the progress of the nation and states’ postsecondary education. As 
Einstein said, "We cannot begin to solve the complex problems we face today with the 
same level of thinking we had when we created them." Hopefully, this conference on post- 
secondary assessment will provide the new level of thinking that will enable the Goals 
Panel to report "the proportion of college graduates who demonstrate an advanced ability to 
think critically, communicate effectively, and solve problems (has increased) substantially." 



THE STATE PERSPECTIVE 



Mark D. Musick, President 
Southern Regional Education Board 

I have been asked to provide a perspective on "state assessment in the current political climate." 
First, let’s acknowledge that there are at least as many different state perspectives as there are 
states but I do have a perspective or a diagnosis, if you will, about assessment in the current po- 
litical climate. My diagnosis regarding collegiate assessment is that the current climate is produc- 
ing schizophrenia or perhaps it is only double vision. My conclusion, however, is that whichever 
of the schizophrenic split personalities you take, or whichever of the double visions you view, 
the conclusion is basically the same — you will see more emphasis, not less, on assessment at the 
college level in the years ahead. 

As a part of the political split personality about higher education, I would argue that there is 
strong public support for higher education and a very strong belief in the value of higher educa- 
tion. I have yet to see a poll that doesn’t show that the American public overwhelmingly believes 
that higher education is important, that parents want their children to continue their education be- 
yond high school, and that Americans believe that education beyond high school is important to 
their future, the future of their children, and the future of the country. Colleges and universities 
remain near the top of the list of institutions in which people have confidence. (Since we are in 
Washington, it would be appropriate, I suppose, to mention that these are the same polls that 
show Congress at the bottom of the list of institutions in which Americans have confidence.) 

Having said that, let’s acknowledge that the level of criticism of higher education among the pub- 
lic, if not at a high point, is certainly high enough to get the attention of nearly anyone who has 
anything to do with higher education. In this age of information overload, the public hears 
weekly about something stupid or at best an action showing questionable judgment that has hap- 
pened on a college campus. And we are not talking about student pranks. I don’t need to recite a 
litany of indiscretions, poor judgments, or in some cases downright fraud, that have occurred on 
our campuses. I don’t know whether there is more or less of these than there used to be, but we 
certainly know more about them than we used to. 

Where the political split personality about higher education comes into play is that there has 
never been more support, certainly not in my lifetime, for assessment at the college level, and 
that support is from the critics and supporters of higher education. This is not simply a Demo- 
cratic/Republican thing. I think an impartial observer would say that Republicans have generally 
been more critical of some of the ways higher education does business and have argued more 
pointedly for greater emphasis on productivity and accountability. We have significantly more 
college-level assessment in the South today than we had ten years ago. That all occurred under 
state houses controlled by Democrats and, for the most part, under Democratic governors. Now, 

I know there is a difference in a Southern Democrat and a national Democrat, but I would argue 
that generally where we have college-level assessment programs, these have had the support of 
the political center. 
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The most fundamental reason that I believe that the current state political climate will produce a 
growing emphasis on postsecondary assessment is that both critics and supporters of higher edu- 
cation are focusing on accountability. Without trying to categorize critics and supporters, think 
with me quickly about some of the things that state leaders are saying. 

Recently in Alabama, the Alabama Commission on Higher Education and the state Board of Edu- 
cation sponsored an Education Summit. The Governor raised a lot of questions about higher edu- 
cation — a lot of open-ended questions. It seemed like he had reached some answers in his own 
mind, chiefly that higher education was getting more money than he thought might be justified 
and that some kind of collegiate assessment was needed. 

A task force in Kentucky was established by the Speaker of the House, who, unquestionably, is a 
friend of higher education. The Speaker established the task force in large measure because he 
felt that higher education was inadequately funded. But one of the initial questions he posed, and 
I quote, is "Kentucky needs a better method for determining the success of our colleges and uni- 
versities in turning out qualified, knowledgeable graduates. How can we devise a better model?" 

The Governor of Arkansas ran on a platform that included reducing remedial education in col- 
lege and of having a rising-junior assessment. 

Colorado Governor Roy Romer, a member of the National Education Goals Panel, has suggested 
that higher education needs a "Consumers Report" or a "J. D. Powers’ survey." (For those here 
not into automobiles, J. D. Powers rates automobiles). Needless to say, Governor Romer’s sug- 
gestions relate to assessment. 

If you read the Chronicle of Higher Education, you may have read about the chairman of a spe- 
cial education study committee in Pennsylvania who was questioning higher education on almost 
all fronts, including its quality. According to the Chronicle article you either think that legislator 
is asking needed questions about institutional priorities and faculty productivity or you believe 
that he is diverting attention from the higher education issues that will help move the state for- 
ward. 

The Wingspread Group on higher education said, "The simple fact is that some faculties and in- 
stitutions certify for graduation too many students who cannot read and write very well and too 
many whose skills are inadequate in the face of the demand of contemporary life." A pretty obvi- 
ous implication is that more or better assessments are needed to see that students do read and 
write well enough and do have adequate skills. 

Public policy institutes are being formed in many states. Generally these institutes have a conser- 
vative agenda and are often focused on education. Most of the education emphasis of these insti- 
tutes has been at the elementary/secondary level. Some have called for greater productivity in 
higher education, have challenged the tenure system, and have called either directly or indirectly 
for more assessment at the collegiate level. 

I think that most people would agree that state higher education boards are supporters of higher 
education. I know that there are persons from institutions that might dispute that. They believe 
that these state higher education boards, be they coordinating or governing boards, are not suffi- 
ciently being advocates for higher education. But the public would see these boards as supporters 
of higher education, and these state higher education boards are increasingly calling for ways to 
ensure quality and to assess student learning. 
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The regional accrediting groups would clearly be seen as supporters of higher education. You 
might debate the intensity or depth of their emphasis on results and outcomes, but clearly the re- 
gional accrediting groups have called for more outcome measures of which a central one is the 
measurement of student achievement. 

The message from the business leadership seems quite clear. Take for example the policy state- 
ment on higher education by the Virginia Chamber of Commerce. The Virginia business leader- 
ship rallied behind a campaign spurred by John T. Hazel, a Northern Virginia business and civic 
leader, who called on legislative candidates to take a pledge in support of higher education. A 
large number did so. The Virginia Chamber of Commerce endorsed a number of goals about 
higher education, and in the first goal it called for "focusing on student needs for knowledge and 
skills by rigorously assessing results." 

Let me modestly suggest that the Southern Regional Education Board’s Commission for Educa- 
tional Quality summarized very succinctly what I believe is the consensus opinion of persons 
who think about higher education and its role in America’s future. The SREB Commission said, 
"Higher education is America’s number one asset. This is an uncertain world. The greater the un- 
certainty about the world and what America may become, the more certain we are of higher edu- 
cation’s role in our future prosperity. We know of the problems in our colleges and universities, 
but we also know of their strengths. The problems we face are twofold. First, state and national 
leaders do not sufficiently recognize the value of higher education in an uncertain world. The 
budget decisions are proof that higher education’s priority is slipping. Secondly, colleges and 
universities do not sufficiently recognize the need to make changes that will keep higher educa- 
tion the number one asset of this nation of free men and women. In a changing nation and world, 
higher education is changing too slowly." One of the ways that the SREB Commission believes 
that higher education is changing too slowly is its emphasis on assessing and reporting its results. 

I think the evidence is abundantly clear that both the critics and supporters of higher education 
will make the need for more effective assessment a part of either their criticism of, or a condition 
of their support for, higher education. 

At least three other specific issues will also lead persons outside of higher education to call for 
more postsecondary assessment. One of these is remedial education in colleges and universities. 

I don’t have to tell this group that the political climate is running in favor of eliminating reme- 
dial/ developmental courses in colleges and universities. And I don’t have to tell you that most 
legislators and governors who discuss the subject cannot fathom how it is that a college uses a 
test to place a student into a remedial/developmental course and then does not require that stu- 
dent to pass the same test to exit from remedial/developmental studies. I look for state policy- 
makers to insist on a more consistent and systematic evaluation of remedial programs. 

The increased use of technology and distance learning will also increase the emphasis on assess- 
ment at the postsecondary level. Whenever distance learning and technology is discussed, either 
the first or second question asked has to do with the quality and integrity of these programs. 
Elected officials want to know, "Do these programs work?" The way they generally want to 
evaluate them is "Do students enrolled in distance learning programs do as well — on tests — as 
students in regular courses? If you can’t answer that question generally in the affirmative, you 
will be out of luck in the distance learning business. Either way, distance learning and the use of 
technology in education will lead to more systematic assessment at the college level. 



133 



137 



I don’t know the effect of a small but growing effort to use college-entry placement tests with 
high school juniors. We now have, on a statewide basis, states that are making available the col- 
lege-placement mathematics or writing tests for use with high school juniors to influence their 
decisions about the courses they take as high school seniors. I don’t know where this will lead us 
in terms of college-level testing, but it will call more attention to this placement testing and draw 
attention to the fact that some institutions have a rising-junior assessment as well as entry-level 
testing. 

Finally, let me underscore my view that assessment must answer questions asked by the public. 
Legislatures and state higher education boards of lay citizens are less prone than educators to "let 
the perfect drive out the good" (or marginally good. ..or possibly good). The argument that a col- 
legiate assessment system doesn’t capture all that is important or measure vital intangible factors 
in higher education will not be seen as a sufficient reason for doing no assessment. Some have 
said that legislators, for example, don’t make policy by relying on precise information and strate- 
gies that are limited and tightly focused. To put it in more descriptive terms, they often don’t 
throw darts when making policies — they throw hand grenades. When you are throwing hand gre- 
nades, getting close can be good enough. A good college-level assessment, not a perfect or near- 
perfect assessment, will be close enough for lay citizens who are looking for answers to basic 
questions that higher education is being asked by the public. 

I don’t see in the political climate today a tidal wave of emphasis on postsecondary assessment. I 
do see a steadily rising interest and a greater emphasis on assessing what college students know 
and can do as well as more efforts to assess the effects of higher education on students’ skills in 
communications, computation, and problem solving. 
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THE VIRGINIA EXPERIENCE 



Margaret A. Miller 

State Council of Higher Education for Virginia 

Virginia was the first state to establish state-mandated, campus-based assessment. In 1986, the 
State Council of Higher Education for Virginia and the Virginia General Assembly thought that 
this approach to assessment could meet the dual goals of program improvement and account- 
ability. Of the two, however, improvement took priority, which influenced a number of decisions 
about the shape of the program. 

• Virginia would not implement state- wide testing, the existing model for assessment at that 
time, and assessment would be not just campus-specific but program-specific. This decision 
was based on the assumption that only when results were tied to specific curricular goals 
would faculty use the results for program improvement. 

• Faculty would own assessment, so that they would take its results seriously. This meant that 
those faculty responsible for programs would choose, tailor, or create the means of measuring 
progress towards their curricular goals. State-level oversight would focus on whether results 
were produced through reasonable and sound methodologies, analyzed, and used to guide ap- 
propriate curricular action. 

• Finally, in order to encourage faculty and institutional commitment to campus-based assess- 
ment, the program was designed to make cross-institutional or even cross-program compari- 
sons impossible. 

Improvement 

The original assumption that a program-based approach to assessment can improve programs 
turned out to be correct, according to the Joint Audit and Review Committee (JLARC) of the Vir- 
ginia General Assembly. JLARC concluded, in a recent review of the Council’s work, that as- 
sessment in Virginia "appears to result in significant curricular reform" (1995 Senate Document 
No. 36). 

The use of assessment results has been chiefly at the level of the majors: there are many exam- 
ples of minor to major curricular revisions, revisions of course sequencing or requirements, an in- 
creased emphasis on certain skills, changed admissions requirements, the introduction of 
integrative experiences (for purposes of assessment, but nevertheless a pedagogically sound 
move), and a developing tendency to see the major as "an overall entity and not simply a random 
collection of individual offerings taught by, and under the control of, individual instructors" (a 
comment from one institution’s restructuring report). 

Moreover, as Virginia’s institutions have struggled with the 1994 General Assembly’s mandate 
to restructure their operations to "effect long-term changes in the deployment of faculty, to en- 
sure the effectiveness of academic offerings, to minimize administrative and instructional costs, 
[and] to prepare for the demands of enrollment increases," assessment has taken on a new func- 
tion. In the first eight years of the program, it was unusual for a campus to use assessment results 
for purposes of planning or resource allocation. However, when asked to evaluate the effects of 
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restructuring, most institutions said that they would rely on their assessment programs for infor- 
mation about whether the changes were being made at the expense of learning. And assessment 
will have another critical role to play in restructuring, insofar as institutions are developing ways 
to award credit not on the basis of seat time but of demonstrated competencies. The maturity of 
assessment has provided the institutions with the evaluation measures needed to undertake such 
experiments responsibly and creditably. 

So far, however, the assessment process itself has had its most profound curricular effect bn gen- 
eral education, paradoxically because the process usually broke down when institutions tried to 
determine what their students had learned in the courses taken outside the major. This failure re- 
vealed a general lack of agreement about what students were supposed to learn from the required 
courses. As one institutional report put it, "The distribution requirements are not organized 
around a set of pedagogical principles; they are designed to ensure that each relevant department 
is represented in the requirements." Since the implementation of assessment, virtually every pub- 
lic institution in Virginia has revised its general-education program, resulting in programs organ- 
ized around learning goals rather than departmental boundaries. 

But it remains to be seen whether the new curricula will be any more susceptible to assessment 
than the old ones were. Some institutions have built promising assessment measures into their 
new curricula, but others have postponed that task. And what even successful assessments of the 
revised general education programs will not tell us is whether Virginia’s students, across the col- 
leges, are developing to a satisfactory degree a few fundamental intellectual skills through the un- 
dergraduate years — for instance, to think critically, communicate effectively, and solve problems. 

Accountability 

Since campus-based assessment has fulfilled its promise to improve programs, it has met one ac- 
countability need: public higher education in the state can assure the public that it is engaged in 
continuous improvement. But the sanguine belief that program-specific, non-comparable results 
would satisfy public information needs turned out to be unfounded, since they are provided with- 
out the larger context that might give that information meaning to those outside the program. Be- 
sides its decontextualized nature, the volume of information that program-based assessment 
generates has been also a problem: for the first eight years, assessment information was delivered 
biennially to the council in documents best measured in inches. And sheer bulk aside, the infor- 
mation on student accomplishment that they contained has been difficult to decipher, even for 
those dedicated to the task. On the basis of those reports no one, presumably not even the writ- 
ers, could answer the global questions the public and policy-makers actually ask, like "How well 
is this. program (or institution, or the higher education system in the state) doing?" Nor does in- . 
cremental, piecemeal change — the results of program-based assessment when it works — make 
for a coherent story. 

Various constituencies of higher education need information about its results for varying pur- 
poses. For instance, students and their parents need information in order to make wise enroll- 
ment choices, while policy-makers need it in order to make decisions about planning and 
resource distribution. In order to meet those needs, rather than reform the original assessment 
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program to make it do a job it was not equipped to do, the Council has proposed and is begin- 
ning to develop a series of publications designed to highlight some indirect indicators of institu- 
tional performance, such as progression and graduation rates. This series of publications is 
designed to answer the questions policy-makers and the public actually have: who enters the pub- 
lic colleges in Virginia, what are their experiences there, how well they succeed, and what hap- 
pens to them after graduation. 

One of the striking results of interviewing the various constituents of higher education about 
what they want to know is discovering the degree to which learning and cognitive development 
are not central concerns for most of them. Parents, for instance, want to know what jobs students 
will get when they graduate and the cost of the credential that is the necessary if not the suffi- 
cient condition of their children’s entry into the middle class. Going to college in Virginia has 
gotten considerably more expensive in the past five years; tuition, always high, has gone up by 
almost a third for in-state students and more than that for out-of-state ones. The steep price in- 
crease has, at its worst, generated a kind of consumerism that is manifested as a desire to know 
how to buy the ticket to the best jobs at the lowest possible price. At its best, it leads to the desire 
to acquire, at the lowest possible cost, the skills and knowledge to do that job, and the next, and 
the next. We would like the parents in Virginia, for instance, to ask the kinds of questions that 
Governor Romer, as the parent of seven children, has been posing to the higher education com- 
munity. He tells about how hard it was to decide where to send them to college, to determine 
where he would get the best return in high-quality learning on his private investment in his chil- 
dren’s future good. 

Legislators generally want to get students through the system as quickly, smoothly, and effi- 
ciently as possible. But they too can ask better of colleges and universities than that. While to 
some legislators the 1 994 restructuring mandate may simply have been an attempt to reduce the 
per-unit cost of educating a student, for others, the quality of that education is of paramount con- 
cern. They are receptive, in other words, to the argument that we should be aiming for learning 
productivity rather than cost-cutting. And the business community reinforces that position when 
it declares its need for graduates with technical skills and general intellectual skills such as com- 
munication or problem-solving, as well as the capacity to work in teams and with people from 
other cultures and subcultures. But if learning productivity can be expressed as learning divided 
by cost, in order to inform policy-makers about what learning is being produced at what cost, we 
need to find some comparable ways to assess that learning directly. 

So the Council staff is beginning to talk, in cooperation with several other SREB states, about 
the possibility of administering a performance-based, general intellectual skills test to a sample 
of graduates across the state. In Virginia we are willing to consider this strategy after almost a 
decade of refusing to do so because there now exists a performance-based but standardized meas- 
ure of such skills. Such an examination would be a third term, of which the other two are pro- 
gram-based assessment and indicators of institutional mission; as a direct but comparable 
measure of learning, it both would fill the hole in the indicators project and could conceivably 
improve programs. 



Such a testing program does, however, entail a series of challenges. First, institutions need to 
agree that the examination would actually permit students to demonstrate intellectual skills that 
they are committed to develop through their curricula and co-curricula. Otherwise we cannot ex- 
pect them to translate the test results into curricular and co-curricular changes that make a differ- 
ence. 

Second, we would need to determine how much of what we anticipate will be the superior skills 
of the seniors at our highly selective colleges and universities, compared to those at the virtually 
open-door institutions, should be attributed to what they have gained from the college experience 
and how much attributed to what they brought with them. 

Then, state bureaucrats would have the daunting task of explaining the difference to the public to 
whom higher education is trying to be accountable. The experience of interpreting the institu- 
tional indicators may foreshadow the difficulty of that task. Despite explanations of the relation- 
ship between progression and graduation rates and the academic and demographic characteristics 
of the incoming classes, for instance, the public tendency was to make simplified judgments that 
did not take those relationships into account. Rather, several unselective institutions were 
thought by some to have failed because they graduated fewer students than their more selective 
counterparts. To the degree that information on student accomplishment is collected and dissemi- 
nated nationally, that problem will become national in scope. 

In Virginia, the higher education community, with the unforeseen but very welcome support of 
the business community, has been successful in convincing state officials that collegiate educa- 
tion is a strategic investment on the part of the government. But now, while we need to continue 
to improve programs, we also need to provide the information necessary to make an informed 
public and private investment, requests for which are echoed in legislatures around the country. 
We hope that higher education in the Commonwealth will be able to sit securely on the three- 
legged stool we have constructed, whose legs are campus-based assessment, indicators of institu- 
tional performance, and a general intellectual skills test. 
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MAKING ASSESSMENT ESSENTIAL WITHOUT 
LEGISLATIVE MANDATES: 

THE MISSOURI EXPERIENCE 

Ava L. Fajen, Charles E. Kupchella, Robert B. Stein 

The current crisis in higher education appears, on a superficial level, to be about adequate fund- 
ing. But it may more accurately reflect doubts legislators and taxpayers are having about the re- 
turn they are receiving on their continuing, already substantial investment. As public pressures 
for accountability increase, state legislatures are in some cases taking matters into their own 
hands (Mahtesian, 1995). Hardly a day goes by without another call for an aggressive reassess- 
ment of higher education to make it more cost effective; some even assert that higher education 
in its present form is outmoded and should be abolished altogether or at least reengineered 
(Gales, 1994). 

All of this comes at a time when many institutions are perceived as proceeding without a coher- 
ent strategic plan to meet the needs of their constituents. Such institutions would receive multiple 
benefits from the development of comprehensive plans for self-assessment to ensure continual 
improvement in serving the interests of their stakeholders (Nyhan and Marlowe, 1995). 

Missouri’s answer to the demand for accountability and assessment, referred to as the Funding 
For Results (FFR) program, is a proactive strategy to head off any need for the imposition of leg- 
islative mandates. FFR operates at both the state level and at the level of individual public institu- 
tions; it seeks to balance statewide needs with institutional goals by rewarding an institution’s 
success in contributing to both (Stein and Fajen, 1995). 

The extremely decentralized and diverse nature of Missouri’s system of higher education brings 
a special challenge to the design and implementation of statewide assessment initiatives. Mis- 
souri is home to over 400 postsecondary institutions, including 12 public community colleges 
and 10 public four-year institutions. While the Missouri Coordinating Board for Higher Educa- 
tion (CBHE) develops and promotes state policies for higher education, it has limited authority 
over the public colleges and universities, each of which is run by a separate governing board. De- 
spite its well-established tradition of local autonomy, however, Missouri has been an active 
player in the national assessment movement since the mid 1980s. 

Missouri’s assessment efforts were led by Northeast Missouri State University’s value-added as- 
sessment program (McClain and Krueger, 1985). Encouraged by the results achieved at North- 
east, state leaders challenged all of Missouri’s public institutions to initiate assessment activities 
which would evolve into campus-wide assessment programs. In the early 1990s, a statewide task 
force established new goals for Missouri’s higher education system regarding efficiency, produc- 
tivity, access, quality, effective governance, and accountability. As a way to document the extent 
to which these goals have been achieved, public institutions submit data annually on a broad set 
of performance indicators. 

During this same time period, Missouri’s Coordinating Board made a commitment to develop a 
statewide performance funding initiative which would integrate budget policies with assessment 
activities. The CBHE first included a performance funding component in the budget recom- men- 
dation for its public four-year institutions in 1991 as the FY 1993 budget was developed. Public 
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two-year institutions were added to the performance funding program in the budget recommenda- 
tion for FY 1995. The performance funding elements that serve as a basis for the FFR budget rec- 
ommendation are a selected subset of the broader set of performance indicators. 

Currently, with support from the Fund for the Improvement of Postsecondary Education 
(FIPSE), Missouri is expanding and refining its performance funding program. While institutions 
continue to be rewarded for their contributions to the achievement of statewide goals, e.g., in- 
creasing the number of minority graduates, they are eligible for additional rewards if they design 
and implement their own mission-based performance funding programs as a way to stimulate 
and document improvements in teaching and learning on their campuses. Thus, Missouri’s cur- 
rent assessment policies encourage institutions to develop programs that address both account- 
ability issues and the strengthening of institutional quality. 

Missouri’s Current Assessment Activities 

In contrast to assessment programs that evolve in isolation^ Missouri is working toward an inte- 
grated assessment program that drives agendas for planning and policy development at both state 
and local levels. The goals of Missouri’s policies on assessment, accountability and performance 
funding include: (1) enhanced understanding of and communication about assessment, account- 
ability and performance funding issues among stakeholder groups; (2) enhancement of state- 
level strategic planning for higher education; (3) reinforcing and expanding institutional plans 
for the improvement of teaching and learning; and (4) strengthened legislative and public sup- 
port of our public institutions. 

To reach these goals, Missouri’s performance funding initiative, Funding for Results (FFR), op- 
erates at two levels. The two tiers of Missouri’s performance funding program are referred to as 
state-level and campus-level. The list of state-level performance funding elements includes some 
that apply to all institutions and some that are sector-specific. Although the state-level perform- 
ance funding elements tend to be oriented toward productivity and efficiency, recent refinement 
of the state-level portion of FFR is producing a gradual shift toward a focus on the measurement 
of quality. In the beginning, Missouri used performance funding to encourage institutions to be- 
gin to assess student performance both in general education and in the major. As institutions 
have increased their commitment to assessing student performance, the amount of reward for 
simply doing assessment has decreased while new rewards linked to the performance of students 
on particular measures have been added to the model. Missouri’s program, however, still pro- 
vides institutions with a small reward simply for engaging in student performance assessments. 
Decisions concerning the selection of assessment instruments remain the prerogative of institu- 
tional personnel. 

When Missouri’s performance funding program first began, it did not make a lot of waves. It in- 
volved a very small percentage of the budget, utilized relatively noncontroversial state-level ele- 
ments, and was proactive (not resulting from a legislative mandate). The state-level portion of 
FFR, which was originally the entire program, received mixed reviews on the campuses. Some 
campuses viewed it as an effective strategy for demonstrating accountability, while others ques- 
tioned its effectiveness. The fact that FFR dollars are not a budget line item, but are rolled into 
the general allocation for each institution, has made its fiscal impact relatively invisible to fac- 
ulty at recipient institutions. 
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In recent years, this initiative has clearly evolved and has gained greater visibility. The definition 
of state-level elements has moved from simple productivity measures toward a greater emphasis 
on measures of quality, the size of the performance funding budget component has increased, 
and now rewards for using performance funding at the campus-level have been introduced. As a 
result of these changes, awareness of the program has increased and assessment and performance 
funding issues have begun to gain more attention from institutional presidents and chancellors, 
key legislators, and the governor’s staff. 

Activities to support the expansion of the FFR initiative have been underway for more than a 
year. By encouraging institutions to incorporate performance funding principles at the campus 
level, this initiative increases ownership and support at the local level. To promote statewide 
ownership of the FFR project and build support among policy makers, groups representing all of 
higher education’s stakeholders have been assembled. In addition, the Missouri Assessment Con- 
sortium, a grass roots organization of assessment coordinators from the public four-year institu- 
tions, has been consulted. All of these groups have met periodically to help shape Missouri’s 
approach to assessment, accountability and performance funding. 

In another significant piece of the FFR effort, a statewide group of faculty and students have 
worked during the last year to define crucial teaching and learning issues and to find ways to 
make the demonstration and reward of teaching and learning improvements a more central focus 
on each campus. The result of their efforts has been published in booklet form and distributed to 
administrators and campus-level performance funding teams at all public institutions. Taken to- 
gether, FFR activities during the last year — extensive efforts to give all constituencies a voice in 
the process, unprecedented statewide communication efforts, and deliberate measures to keep 
teaching and learning issues in the spotlight — have all focused on promoting institutional suc- 
cess in creating measurable improvements in teaching and learning. 

Missouri’s governor and legislature have evidenced comfort in using performance funding in the 
allocation process without the heavy hand of a legislative mandate. This initiative is inten- tion- 
ally structured so that it will not drastically affect the overall budget of an institution in any 
given year. The following table provides both recommended and allocated amounts of the 
budget from FY 1993 forward for public two- and four-year institutions. 
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Requested vs. Appropriated Dollars and Percentages 



Public Two-year Institutions 



Fiscal 

Year 


Dollars 

Requested 


Percent* 

Requested 


Dollars 

Appropriated 


Percent 

Appropriated 


1993 


- 


- 


— 





1994 










1995 


1 .5 million 


1.5 


500,000 


.6 


1996 


1 .7 million 


1.8 


799,690 


.9 


1997 


2.5 million 


2.9 










Public Four-year Institutions 




Fiscal 

Year 

1993 


Dollars 

Requested 

6.8 million 


Percent* 

Requested 

1 


Dollars 

Appropriated 

0 


Percent* 

Appropriated 

0 


1994 


6.7 million 


1 


3,021,849 


.5 


1995 


10.0 million 


2 


4,272,131 


.8 


1996 


13.2 million 


2.2 


7,010,000 


1.2 


1997 


13.5 million 


2.4 


** 


** 



* Percent of total core general revenue operating budget 
** Currently under review 

While the amount of money associated with performance funding remains relatively small, i.e., 
less than 3 percent of the total state allocation for either sector, it is large enough to have an im- 
pact. For FY 1 997, more than $16 million are included in the performance funding portions of 
the CBHE budget recommendation for higher education. This represents approximately 19 per- 
cent of the new money recommended for four-year institutions and 13 percent of the new money 
recommended for two-year institutions. Each year has seen a slight increase in the amount recom- 
mended as well as the amount allocated. The message to institutions is clear; performance fund- 
ing in Missouri is on the rise. 

Campus-Level Performance Funding Projects. The campus-level tier of FFR is in its pilot 
year. Each campus was eligible to be recommended for performance funding rewards in the next 
budget cycle if it submitted a plan for a pilot FFR project that identified a major teaching/learn- 
ing issue. Although the submission of a campus-level pilot project for the 1995/1996 year was 
voluntary, all 29 public institutions chose to participate. The proposals for the pilot projects were 
reviewed by statewide groups of institutional representatives and by CBHE staff. 
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The submitted projects reflect differences in focus and scope, but all address specific teach- 
ing/learning issues. Each project has the support of the institution’s president and/or chancellor, 
involves a campus-level FFR team composed of both administrators and faculty, and links a por- 
tion of the institution’s operating budget to rewarding performance. All projects involve the use 
of defined intervention strategies, the collection of baseline data, and an evaluation plan. 

The goals of the proposals range from improving the basic study skills of community college stu- 
dents to enhancing the residence experience of doctoral students. While some projects directly 
address ways to improve the performance of students on assessments, others focus on enhance- 
ments for student advising or on easing the transition from the community college environment 
to the four-year institution. 

In a few cases, institutions have chosen to develop an open process whereby all units are invited 
to submit proposals that address a teaching/learning issue within the mission framework of the in- 
stitution. This approach has the advantage of integrating performance funding into an ongoing 
strategic planning process. In such instances, Funding for Results is not perceived as a new and 
separate initiative; rather it is used to re-energize an existing infrastructure. Initial reports suggest 
that this approach has been very successful; many more proposals are being submitted than can 
be funded. 

State policy makers are beginning to perceive Funding for Results as a catalyst for change. On 
some campuses, faculty are perceiving renewed support for their teaching roles, and an increased 
level of interaction with their administrators. Although demonstrating a rich diversity of ap- 
proaches, each of the campus-level projects now underway reflects a significant institutional 
commitment to the improvement of teaching and learning, and reflects an understanding of the 
importance of using data as a basis for decision making and policy development. 

An Institutional Perspective: Southeast Missouri State University 

The experimental development of a campus-level Funding for Results (FFR) program came at a 
very opportune time for Southeast Missouri State University (SEMSU). The campus was in- 
volved in a comprehensive strategic planning process as the campus-level FFR idea began to 
take shape. Strategic plans were under development within each academic unit and at all levels 
within the organization. One of six strategic priorities identified early in the process was "Offer a 
top-quality curriculum with a solid liberal education as a foundation for preparing graduates for 
leadership positions in society." Another priority focused on "enhancing learning." Each unit 
was asked to indicate how it would go about addressing the university’s overall priorities. Each 
academic unit was also asked to assess the general effectiveness of its curricula and to indicate 
how this effectiveness is/will be determined on an ongoing basis. 

As the University Planning Committee worked to develop campus "buy-in" of the strategic plan- 
ning process, it became clear that one of the historical sources of skepticism about the effective- 
ness of planning on the campus was that planning never seemed to be connected to budgeting. 
The University Planning Committee realized that if planning were to become an effective way of 
life on the campus, faculty would need to know that planning and budgeting are indeed con- 
nected. The timing of the FFR program afforded the campus an opportunity to build a planning- 
budgeting connection into the early stages of its major strategic planning initiative. 

A second aspect of the timeliness of FFR had to do with assessment. The campus received ap- 
proval of its institutional assessment plan from its regional accrediting body in the spring of 
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1 995 and the campus was moving through the second phase of refinement of its assessment pro- 
gram. At the time campus-level FFR began to take form, SEMSU was beginning to connect stra- 
tegic planning, program assessment, and continuous quality improvement. FFR was seen as a 
way to accelerate the use of assessment as a means of driving continuous improvement. 

SEMSU’s approach to the state challenge of developing and implementing a campus-based FFR 
pilot program was to design a process whereby all units could have an opportunity to get in- 
volved. As a result of a team effort, faculty, departments, and other units were invited to offer 
their strategic objectives having to do with teaching and learning enhancement for possible fundr 
ing. The pilot program was funded by reallocating a portion of the institutional/campus budget. 
The review and funding of FFR initiatives was scheduled to occur immediately after the units 
had completed their strategic plans. 

The program was designed and the reviews were carried out by a "Funding for Results Team." 
This fourteen-person team is made up of members of both the University Planning Committee 
and the University Budget Review Committee, providing yet another connection between budg- 
eting and planning. The review was set up as a grant proposal competition using a simple format 
with a two-page limit for the narrative. Clear criteria for evaluation were spelled out by the FFR 
team. These included 1) degree of direct focus on improvements in teaching and learning, 2) clar- 
ity of definition and measurability of outcomes, and 3) clarity of the relationship of the project to 
the unit’s strategic plan. 

Although expectations were high, SEMSU’s campus-based FFR team was overwhelmed to re- 
ceive sixty-nine proposals. These included a wide variety of proposed projects seeking to en- 
hance oral communication skills, written communication skills, technological communication 
skills, problem-solving skills, critical thinking skills, and student success in general. Some exam- 
ples will help illustrate the variety and scope of the projects. The Department of Speech Commu- 
nication and Theater proposed to develop and test oral communication modules for selected 
courses in other majors. Improved speech communication would become one of the curricular 
objectives in the selected courses. Gain would be assessed subjectively by the course instructors 
and by the students, themselves, through course evaluations. The principal assessment measure, 
however, would be an instrument developed by the Speech Department. Ultimately, this "mod- 
ule" idea would be developed as a "speaking across the curriculum" program, university wide. 

The Departments of Criminal Justice, Industrial Technology, and English, (and several other de- 
partments) proposed projects to enhance student writing. The Department of Criminal Justice 
proposed to identify students likely to have difficulty passing the university’s writing proficiency 
exam and direct them to a special writing workshop. Industrial Technology proposed a series of 
workshops to help faculty teach writing about disciplinary subjects more effectively. This pro- 
ject’s effectiveness would be assessed by measuring the number and richness of writing assign- 
ments by targeted faculty before and after the workshop. The Department of English proposed to 
give faculty members computer upgrades equivalent to computers in a new "computer composi- 
tion" laboratory. Effectiveness would be assessed via number and type of composition lab assign- 
ments given to students by faculty. Currently, SEMSU has a university-wide writing assessment 
program. Students are required to pass a writing exam (with a highly developed, standardized 
scoring rubric) after completing 75 hours of course work. All of the FFR writing enhancement pi- 
lot projects plan to use baseline and outcome data collected by the university’s writing assess- 
ment program. 





Four departments proposed programs to develop student technological communication skills. 

The development of skills such as the effective use of multimedia in making presentations would 
be assessed by asking the students to demonstrate the skills. Several departments proposed pro- 
jects that would involve students in "real-world" problem-solving activities to enhance critical 
thinking skills. Industrial Technology seniors would be given problems actually being experi- 
enced in regional manufacturing industries. The students, in small groups with a faculty mentor, 
would tackle the problem and propose a solution. This learning strategy was presented as an "ulti- 
mate" test of the student’s ability to solve problems as members of a work team. Sociology and 
Anthropology, Biology, Criminal Justice, Psychology, and other departments proposed quality 
improvements in internship programs with similar intentions of engaging students in real world 
problem solving situations. 

There were many proposals to enhance student success ranging from instituting daily quizzes in 
key business courses to providing various forms of tutorials for students in math, accounting, 
and other "difficult" courses. The success of the interventions would be measured against histori- 
cal baseline data or, in some cases, "control" classes not receiving the intervention. Also pro- 
posed were a large number of teaching enhancements mostly having to do with the use of 
information technology. There were twenty-three proposals asking for computers or computer-re- 
lated hardware and software. Reviews are still in progress at this time; the FFR Team has been 
impressed by the breadth and depth of ideas that have been generated. 

Although budget constraints will preclude the funding of all worthy projects, it is clear that as a 
result of our FFR pilot project many exciting ideas are germinating on campus. Budget deci- 
sions, not just involving our FFR allocation, but those that are part of the larger budget process 
on campus will be influenced. SEMSU is succeeding in using assessment to make improve- 
ments. In addition, reward structures are being realigned so that faculty are more integrally in- 
volved in shaping the institution’s future. Teaching/leaming issues are becoming a central focus 
in the budget and planning process of the university. 

State-Level Perspective on Funding for Results 

From the perspective of state policy makers, Funding for Results has elevated the discussion 
about the needs of higher education from a litany of complaints concerning a lack of resources to 
a focus on performance and institutional achievements. While resources remain a major concern, 
conversations about opportunities, constraints and accomplishments are moving toward a data- 
driven rather than an anecdotal basis. 

Requiring institutions to systematically identify and analyze outcomes across a wide array of ele- 
ments has a number of positive effects. Institutions are forced to engage in self-reflection about 
effectiveness and value using agreed-upon criteria. Baseline data are established so that improve- 
ments can be tracked over time. When visioning for the future, institutions are pressed to identify 
measurements that will clearly demonstrate achievement of specific goals. Institutions are also 
better positioned to ensure continual improvement by systematically using assessment informa- 
tion to evaluate their progress. Last but certainly not least, as accomplishments are documented, 
higher education’s communication with external publics can be greatly enhanced. 

As an initiative for change, performance funding helps to focus attention on assessment. By link- 
ing a small portion of budgets to results at both the state and campus levels, i.e., state appropria- 
tions to institutions, and internal appropriations within institutions, both state level priorities and 
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institutional aspirations are made salient. A complex assessment program can be made to serve 
multiple purposes, i.e., accountability to external publics and improvement of teaching/learning 
on campus, simultaneously. When linked with assessment and accountability activities, perform- 
ance funding can help to narrow the difference between faculty perspectives and those of state 
policy makers. 

At the state level, the distribution of a limited pool of funds among the higher education institu- 
tions is being reexamined. Institutions contribute differentially to state priorities and should be re- 
warded accordingly. A key issue then is the setting of state-level priorities and how assessment 
results should be used to demonstrate institutional responsiveness to state interests. In addition, 
the total amount of money allocated by means of performance funding should remain small 
enough that changes in institutional allocations from one year to the next are not too great. 

At the campus level, the majority of funds within colleges and universities are committed years 
in advance, e.g., personnel costs. The expenditure of contingency money, therefore, takes on ex- 
tra symbolic significance. Traditionally the distribution of any flexible portion of university and 
college budgets has been based on historical precedent. The use of a portion of the budget to re- 
ward results thus requires a paradigm shift. 

Suddenly questions about what to reward, when to reward, and who should receive a reward be- 
come important considerations. Initial approaches might define reward structures in simplistic 
terms, i.e., those most responsible for bringing about a desired change receive a reward. Merit 
pay systems represent this traditional approach to performance funding. The FFR initiative, how- 
ever, is resulting in a broadening of the definition of reward structures through a greater aware- 
ness and involvement by personnel across the campus. 

From the outset, Missouri has been committed to building a performance funding model that will 
make a positive difference in the everyday lives of faculty and students. FFR is building a state- 
wide coalition for change that links public policies on funding, assessment and accountability 
with programs to improve teaching and learning. Through its performance funding initiative, 
Missouri continues to raise the stakes associated with traditional assessment activities and has en- 
sured the centrality of assessment on the campuses of its public institutions. 
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Methodological Concerns: An ETS Perspective 

John Mazzeo 

Educational Testing Service 

As a participant in this session on Methodological Concerns for the 1995 NCES State Postsecon- 
dary Assessment Activities Workshop, I have been given two charges. The first is to say a little 
about Educational Testing Service’s (ETS) general approach to postsecondary outcomes assess- 
ment. The second is to discuss some of the lessons learned from the National Assessment of Edu- 
cational Progress (NAEP) that might be relevant to the post-secondary assessment context. 
Section I of this paper discusses some existing ETS assessment instruments that are appropriate 
for postsecondary outcomes and the general assessment philosophy underlying the creation of 
these instruments. Section II of the paper discusses some lessons learned from NAEP that may 
be relevant to the postsecondary assessment context. 

Section I. ETS’ general approach to postsecondary outcomes assessment and the instru- 
ments and services it currently provides 

In general it is ETS’ belief that the use of any test or instrument should be one part of an overall 
and ongoing assessment effort. Assessment efforts should include: instructor-written classroom 
tests, portfolios of student work, measures of student achievement in the major, department/pro- 
gram reviews, alumni follow-up and student evaluation of courses and instruction. 

The choice of instrument should be determined by the kinds of information that will be most use- 
ful to faculty and administrators. Assessment is not an end in itself; it is a way of obtaining infor- 
mation to help focus on and understand problems. Results should suggest actions that will lead 
to the improvement of teaching and learning. 

ETS offers a variety of tests, instruments and services that can be used in outcomes assessment 
and institutional evaluation. These offerings are intended to supplement local measures (as well 
as other external measures) and to enhance total assessment efforts. 

Academic Profile — A comprehensive measure of the academic skills acquired through un- 
dergraduate general education courses. 

The test measures: college-level reading, college-level writing, mathematics, and critical think- 
ing in the context of material from the humanities, social sciences, and natural sciences. Scores 
are reported in terms of proficiency levels (in writing, mathematics and reading/critical thinking) 
and as standard scale scores in the four skills and three academic areas noted. 

Tests consist entirely of multiple-choice items and two forms are available: (1) a short form (45- 
minutes) which provides an overall student score and institution-level scores in the four skill and 
three academic areas described, and (2) a long form (2.5 hours) which provides individual and in- 
stitution-level scores for the 12 skill/area combinations. Institutions can tailor the test by adding 
up to 50 locally written questions to the test. 

Tasks in Critical Thinking — a performance-based measure of student abilities to use higher- 
order or critical thinking skills. 
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Using an extended response format, the tests set a problem and direct the student through various 
steps necessary to solve it. Tests were designed by college faculty and were intended to resemble 
what students are required to do in the classroom and the world of work. Each task is set in the 
context of one of three broad academic areas (humanities, social sciences or natural sciences) 
and consists of 8 or 9 short answer questions followed by an essay or report. 

Scores are reported for groups (not individuals) as the percent demonstrating proficiency in each 
of three skill areas — inquiry, analysis, and communication. Scoring can be done entirely by local 
faculty (using detailed materials developed for each test), using local faculty but directed by an 
ETS consultant, or entirely by ETS. 

Major Field Tests — multiple-choice tests that assess mastery of concepts, principles, and knowl- 
edge typically expected of students upon completion of undergraduate major in a given subject. 

These tests are available in 14 disciplines. Score reports include a departmental summary show- 
ing total score and subscores for each student and average scores for the department. In addition, 
two to eight assessment indicators (scores that relate to subfields within a major discipline) are 
reported for groups of tested individuals (but not at the student level). National comparative data 
are available for each test. Departments may add locally written questions that address depart- 
mental concerns. 

Participants interested in more information about any of the above products should contact 
Nancy Beck of ETS at 609-92 1 -9000. 

Section II. What lessons have been learned from NAEP? 

Stakeholder consensus is a must 

Consensus and stakeholder involvement are extremely important from the outset. Ideally, stu- 
dents, faculty, university administrators, state policy and legislative officials, the business com- 
munity, and the public at large should have an opportunity to communicate their ideas and 
concerns openly and aggressively as assessment systems are being established. Agreement needs 
to be sought regarding: i) assessment purposes, ii) the identification of pertinent skills/knowl- 
edge to be assessed, iii) assessment formats, iv) survey designs, v) performance standards, and 
vi) reporting formats. 

Our experience with state NAEP suggests to us that a major contributor to the success of state 
NAEP has been keeping state assessment officers and curriculum staff actively involved in the 
design and conduct of NAEP. The curriculum frameworks on which the assessments are based 
have been developed (under NAGB and CCSSO auspices) using a concensus process in which 
state curriculum experts have had opportunity to actively contribute. The NAEP instruments con- 
structed from those frameworks are reviewed (at the field test and final form stages) by desig- 
nated state curriculum staff and their input discussed and, where appropriate, incorporated. 
Regular meetings of "THE NETWORK" (which consists of state assessment directors and cur- 
riculum staff) are held (3 times a year) and provide an opportunity for state input on all aspects 
of the program (minimum participation quidelines, administration and monitoring procedures, re- 
porting formats, plans for press releases). 

Of course, talking about getting concensus and actually obtaining consensus (particularly with re- 
spect to assessment frameworks and instruments) are two different things. There are a variety of 
things that can go wrong. One way in which concensus can backfire is in producing an instru- 



ment that represents the lowest common denominator among the constituencies. Another prob- 
lem, (more often the case in our experience with NAEP) has been a tendency for panels with di- 
verse viewpoints to produce an instrument that is overly ambitious. Getting broad based panels 
to acknowledge the realities of cost constraints and administrative realities is not always easy. At 
a minimum, the concensus process needs to be iterative involving the preparation of multiple 
drafts of planning documents with mechanisms after each draft for eliciting and incorporating 
stakeholder comments. 

Assessments must be perceived as improvement oriented 

Our NAEP experience is consistent with the view that the greatest likelihood of success will 
acrue to assessment efforts that are seen as improvement oriented. Institutional performance data 
should be accompanied by contextual data about the student populations being assessed, their op- 
portunities to learn the assessment materials, and the resources available to faculties and institu- 
tions in support of learning. Reducing the emphasis on the "horse race" (i.e., comparisons across 
institutions, or departments within the institution) and focusing more on institutional improve- 
ment overtime is in the best interests of everyone involved. 

In my opinion, NAEP’s pre-eminent purpose should be the reporting of credible trend data. It 
provides one mechanism for states and the nation to monitor their movement toward instruc- 
tional improvements and achieving of educational goals. Trend results are treated prominently in 
both national and state reports. 

Furthermore, NAEP collects background data on student demographic characteristics, student 
self-reports on instructional experiences, study habits, home support for schooling activites; 
teacher reports on their levels of schooling, certification, professional development, and teaching 
activities; school reports on programs, policies, and school climate variables. An attempt is made 
to focus attention on interpreting performance data (and any changes or improvements in per- 
formance) in light of these background variables. Such data also allow for identifying potential 
areas of improvement in the delivery of educational services. 

Perhaps the most useful document for states that participate in NAEP is an individualized report 
on their states’ test results and the background data collected. While each state’s report does con- 
tain sections comparing the state-level test performance to that of the nation, a large portion of 
the report is focussed on what instructional activities are going on in the classroom, the training 
and preparation of the teachers delivering instruction, what students are doing with their time out- 
side of school, and home support activities for learning. My experience in talking to state testing 
directors is that this latter information is far more useful to them in talking to schools, policy ana- 
lysts, and legislatures within the state about how education might be improved. 

A common metric for comparing performance across jurisdictions will have appeal 

Most institutions (and jurisdictions) view their goals, their curricula, the problems they face and 
the populations they serve as somewhat unique. They at least initially fear cross-institution com- 
parisons and the inappropriate (and potentially punitive) consequences associated with (what 
they see as) inappropriate inferences about the quality of schooling at their institution. 

Growing interest in state NAEP suggests that if a common yardstick can be demonstrated to be 
credible and fair (and if the first two points above are successfully achieved), institutions may 
come to view such results as highly valuable. At NAEP’s inception (over 20 years ago) state- 
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level results were politically unthinkable. By the late 80’s, the states themselves were requesting 
specific state-level results. In 1990, the first year of state NAEP, 40 jurisdictions participated in 
this voluntary program. The numbers rose to 44 for both the 1992 and 1994 assessment cycles. 
The number is 50 for the 1996 state assessment (the third such assessment). Furthermore, there is 
now interest on the part of some of the larger school districts around the country (Philadelphia, 
Milwaukee, Atlanta, to name a few). 

It seems to me that the many state educators, legislatures and policy staff have come to value the 
comparative "benchmark data" (national and state comparisons) and the ongoing high quality 
normative results that the program provides. The 1996 assessment will include an attempt to link 
results to the TIMMS (an international science and mathematics assessment). If successful, such 
linking would allow for the possibilities of comparing results to a large number of countries. In 
my opinion, this has resulted in a further increase in participation in state NAEP. 

Sampling versus all student testing 

Unless one is testing for student certification purposes, sampling-based assessments offer consid- 
erable advantages. There are two types of sampling to distinguish here: examinee sampling and 
item sampling. With examinee sampling, a typically small proportion of the school/student popu- 
lation is tested and the intent is to generalize the results to the entire population of schools/stu- 
dents. With item sampling, any particular assessed student is administered only a portion of the 
pool of assessment tasks. Combining these two approaches yields a matrix sample — one first 
samples a group of students, divides the sample fiirther into subgroups, and administers each sub- 
group a portion of the pool of assessment tasks. Matrix sampling approaches are what are used in 
NAEP. Matrix sample approaches to assessment are highly efficient in that they minimize stu- 
dent testing burden while allowing for a broad representation of the item pool. 

When confidentiality of participants within a jurisdiction (institution) is assured, sampling-based 
approaches tend to minimize concerns about cheating or inappropriate kinds of "teaching to the 
test." However, a potential problem with sampling-based approaches concerns the motivation of 
the tested individuals. Why should an individual in such an assessment work hard to do their best 
if there is nothing at stake for the individual? Motivational issues are particularly salient in situ- 
ations where standards-based reporting is in use. Results from assessments are sometimes ex- 
pressed in terms of the proportions of students whose test results exceed some pre-established 
standard. However, the standard is often set without considering that portions of the assessed 
population may not be trying their hardest on the test. Such considerations are also confounded 
by the level of effort demanded by the task. If the tasks involve considerable student effort, the 
performance of unmotivated students may look unexpectedly low. 

The evidence from NAEP on the effects of motivation (or lack thereof) have been mixed. Re- 
search by O’Neill et al. using NAEP math tests showed that financial incentives for performance 
($1 per each correctly answered item) raised test scores by 1/2 standard deviation (sd) among 8th 
graders but did not do so for 12th graders. Work by Kiplinger and Linn, in which two sets of 

NAEP items (which were administered to students under low stakes situations for the stu- 
dents/schools) were embedded within a high stakes state testing program. Higher performance in 
the high stakes context was found for only one of the two item sets. But clearly, taken as a 
whole, these studies highlight a weakness of sample-based low-stakes testing. 



If an examinee sample-based approach is chosen, such an approach can be carried out in terms of 
a cross-sectional or a longitudinal design, e.g., NAEP is a cross-sectional design. Every 4-years 
different representative samples of 4th, 8th, & 12th graders are tested. The sample of 8th graders 
tested in 1994 is not the same individuals who were tested in 1990 (as 4th graders). In a longitu- 
dinal design, the individuals tested in 1990 (as 4th graders) would be tested again in 1994 (as 8th 
graders) and again in 1998 (as 12th graders). Each of the two sample designs has pros and cons 
associated with it. 

Cross-sectional versus longitudinal designs 

Cross-sectional designs are, in some sense, simpler to deal with. An individual is tested once and 
no follow-up is required. The cross-sectional design is especially amenable to a group-level fo- 
cus describing the aggregate achievement of the current population and, as such, efficient de- 
signs (like matrix samples) work well within this context. There are, however, some clear 
limitations. It is often tempting to interpret across cohort comparisons (e.g., 4th and 8th graders 
at a given time point of 8th graders in 1994 vs 4th graders in 1990) as showing the amount of 
growth in test score performance over the indicated period. But such interpretations are con- 
founded by other factors. Comparing 8th graders and 4th graders in a given assessment year con- 
founds cohort differences (in, say demographics and SES) with growth. Comparisons of the 
same cohort (e.g., 12th graders in 1994 with 8th graders in 1990) ignores the fact that drop-out 
occurs and the drop-out rate can be expected to be correlated with level of achievement. Another 
limitation of the cross-sectional design is that it is next to impossible to establish convincing 
"causal relationships" between process and product variables. In cross-sectional surveys, correla- 
tional patterns can be observed and established but "correlation does not imply causation." 

Longitudinal designs can be difficult to deal with in certain contexts since the individuals in- 
itially tested must be traced and followed up with later testing. (This may be less difficult in the 
postsecondary school context than in other settings but will probably still cause problems.) In a 

longitudinal design, the emphasis is on getting reliable individual measures of growth and aggre- 
gating these to describe growth at the group level. This generally involves tests that are reliable 
at the level of individual scores and hence such designs are not very amenable to the efficient ma- 
trix sample designs discussed earlier. In particular, domain coverage can be a problem unless 
each individual is tested for a fairly long period of time. There are, however, some clear advan- 
tages to these designs. The initial testing and follow-up of the same individuals and the presence 
of individual student scores make the provision of student feedback possible. This may have de- 
sirable effects on increasing student motivation. Furthermore, if growth is the focus, the cohort 
effects which confound interpretations in a cross-sectional design are mitigated (since growth 
measures at the level of the individual are obtained). If one is interested in relating process and 
product variables, time sequencing information (and a variety of well developed data analytic 
techniques) make this easier to do in a longitudinal design. 

Some Concluding Remarks 

In concluding my comments for this session, I feel it only fair to note a few things about my 
background since they no doubt color my views on the feasibility of broad-based postsecondary 
outcomes assessment. My training and background is in the area of psychometrics and educa- 
tional statistics. Throughout my career, my role on the various assessment projects that I have 
been involved with has been primarily that of methodologist and data analyst. I am not a test de- 
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veloper (i.e., I do not write items), nor am I involved in educational administration or policy 
analysis (i.e., I have relatively little to do with the politics of educational assessment activities). 
None of the projects I have worked on have involved postsecondary assessment. Therefore, I 
hope the session participants will excuse my naivete’ as to the peculiar assessment challenges 
that arise in the postsecondary arena and my focus on methodological rather than substantive 
concerns. 

In recent years, I have worked almost entirely on the State Assessment component of NAEP. As 
most (if not all) of you know, NAEP is a sample-based assessment that is designed to provide 
group-level information on the educational achievement of America’s in-school student popula- 
tions at grades 4, 8, and 12. Beginning in 1990, group-level data has been provided on a state-by- 
state basis to jurisdictions that participate on a voluntary basis. In my view, NAEP is an 
extremely important educational indicator and the conduct of a voluntary state NAEP has been 
successfully accomplished. In short, I am a believer in the feasibility and utility of centralized, 
standardized, sample-based assessments like NAEP for the monitoring of educational progress 
and think that this model represents an attractive alternative for cross-state assessments of post- 
secondary outcomes assessment. 

I am quite grateful for having had the opportunity to work on NAEP. During my tenure of in- 
volvement with NAEP it has been carried out in an atmosphere of general political support and 
with a great deal of economic resources available. Because of this environment, NAEP has had 
the luxury to design and execute a broad-based cross-state assessment project with a great deal 
of scientific rigor using what I believe to be cutting-edge assessment, methodological and data 
analytic methods. What I have learned about carrying out such assessments I have learned in this 
ideal environment. As a result, my views on the prospects of the NAEP model for postsecondary 
assessment are no doubt optimistic — many of you no doubt judge them as overly optimistic. 
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